Add missing docs, tools, and scripts

This commit is contained in:
Jorrit Jongma 2013-12-06 15:08:22 +01:00
parent b4c95536bc
commit 73fcb1546d
4 changed files with 482 additions and 2 deletions

140
README.md
View File

@ -1,3 +1,139 @@
Pushing the sources first, will update this file ASAP
About
-----
- Chainfire
**OpenDelta** was written to provide automatic OTA updates for
**The OmniROM Project**'s nightly builds, making use of deltas when possible,
to reduce the size of the download.
There's no reason you couldn't use it for weeklies or monthlies as well though!
License
-------
**OpenDelta** is licensed under the terms of the *GNU General Public License,
version 3.0*. See the *COPYING* file for the full license text.
How
---
**OpenDelta** uses binary differentials (VCDIFF, RFC 3284) between the previous
and the current release, courtesy of **xdelta3** (<http://xdelta.org/>).
Usually, OTA ZIP files created by Android builds are compressed. Diffs between
compressed OTA ZIPs are not ideal - they are significantly larger than the diff
between two uncompressed OTA ZIPs would be. So before we create the diff we
decompress the contents of the OTA ZIPs.
The whole-file signature of the OTA ZIP is broken (and removed) by this process,
and so we also re-sign the decompressed ZIPs with the same keys used to build
Android. We create a second diff between the unsigned and re-signed ZIP file,
so if needed the client can re-create a properly signed ZIP file.
At the time of this writing, that signature is not actually used. This is
because **OmniROM** doesn't use the public platform test keys, but private keys
specific to **OmniROM**. The recovery running on your device will most likely
not be built with these keys, and thus whole-file signature checking will
fail anyway (if enabled). So to save a bit of processing, this feature is
turned off by default. The needed files are generated and the client knows how
to deal with them, so enabling this feature is just a configuration switch away.
The produced delta files are pushed to the public download server, and the
current build is saved to a private location to serve as input for the next
differential run.
It is important to note that the differential files are named after the *input*
file, not after the *output* file. Initially this may seem a bit confusing when
working with these files, but this way the client doesn't need to know any
information about future builds when looking for updates, and no server logic
is needed at all - just a public download location - as the delta filename can
be reconstructed from getprop's on the device.
The Android client periodically checks in with the download server and
retrieves the *.delta* file for its current build. After parsing it, it knows
the name for the next build as well, and then the one after that, etc. So
if you don't update for a number of builds, it can still reconstruct the latest
build by chaining the deltas. It will check each delta if we already have
intermediate files present - perhaps we already performed the work for the last
build but never flashed it, for example. Based on all this information it will
device to either reconstruct the final flashable ZIP, or just download the
latest full OTA and flash that.
Flashing is currently tested only against **TWRP**.
Bad builds
----------
As **OpenDelta** depends on an unbroken chain of deltas, you can't just remove
the files of a bad/dangerous/etc build. If you want to prevent the client from
producing and flashing such a build, rename the relevant *.delta* file to
*.delta_revoked*.
We'd still have a problem if you want to produce a replacement build, or for
some reason have several different builds with the same name, and this is
breaking the chain of deltas. The solution for this is to edit the relevant
*.delta* file, and setting the *size* of the *update* file to a value larger
than the *size_official* of the *out* file. This will trigger the client to
download the full-size compressed OTA ZIP instead.
Server-side
-----------
The create the delta files on the server, you need several things, some of
which can be found in the *server* directory. The main thing is the
*opendelta.sh* script. It contains a configuration section which you can edit
with the correct file locations on your own system. Quite likely you will need
to create a wrapper script that pulls in your previous release and your
current release, and pushes out the created delta files.
The script depends on *xdelta3*, *zipadjust* and *minsignapk*.
For the builds on your server, make a *copy* of the *jni* directory - do **not**
compile inside *jni* because you may mess up the build of *libopendelta*.
*xdelta3* can be built in (the copy of) *jni/xdelta3-3.0.7* by calling *./configure*
and *make*.
*zipadjust* can be built in (the copy of) *jni* by calling:
gcc -o zipadjust zipadjust.c zipadjust_run.c -lz
*dedelta* (not used by the script, but maybe helpful when debugging) can be built
in (the copy of) *jni* by calling:
gcc -o dedelta xdelta3-3.0.7/xdelta3.c delta.c delta_run.c
*minsignapk* Java source is in the *server* directory, as well as a prebuilt
*minsignapk.jar* file that should work on most systems
Eclipse
-------
For debugging purposes you may wish to build in Eclipse instead of an Android
tree, for test-speed benefits. The native part of **OpenDelta** is also NDK
buildable.
You may need to enable the app to show up in the launcher ("System Updates")
by editing *AndroidManifest*.
The APK needs privileged system permissions, and thus needs to placed in
*/system/priv-app*. If you're testing on a build that includes **OpenDelta*
already, remove it from that location and reboot before continuing. If you
install the APK through Eclipse it'll end up in */data/app*, but will not be
granted the right permissions. Move that APK to */system/priv-app* and reboot.
Now increase the *versionCode* in *AndroidManifest* to a larger number, and
your Eclipse-installed builds will magically run with the right permissions
granted every time. You could use *pm grant* but you'd have to do that after
every install.
Aside from inside the APK, you also need to place the produced *libopendelta.so*
for your architecture in */system/lib*. If you're actually working on the
native library this gets annoying fast, symlinking that location to the library
location from the APK can save you a lot of headache.
-EOF-

191
server/MinSignAPK.java Normal file
View File

@ -0,0 +1,191 @@
/*
* Copyright (C) 2008 The Android Open Source Project
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
/* This is just a copy/paste/cut job from original SignAPK sources. This
* adaptation adds only the whole-file signature to a ZIP(jar,apk) file, and
* doesn't do any of the per-file signing, creating manifests, etc. This is
* useful when you've changed the structure itself of an existing (signed!)
* ZIP file, but the extracted contents are still identical. Using
* the normal SignAPK may re-arrange other things inside the ZIP, which may
* be unwanted behavior. This version only changes the ZIP's tail and keeps
* the rest the same - CF
*/
package eu.chainfire.minsignapk;
import java.io.ByteArrayOutputStream;
import java.io.DataInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.security.GeneralSecurityException;
import java.security.KeyFactory;
import java.security.PrivateKey;
import java.security.Signature;
import java.security.cert.CertificateFactory;
import java.security.cert.X509Certificate;
import java.security.spec.InvalidKeySpecException;
import java.security.spec.KeySpec;
import java.security.spec.PKCS8EncodedKeySpec;
import sun.security.pkcs.ContentInfo;
import sun.security.pkcs.PKCS7;
import sun.security.pkcs.SignerInfo;
import sun.security.x509.AlgorithmId;
import sun.security.x509.X500Name;
public class MinSignAPK {
/** Write a .RSA file with a digital signature. */
private static void writeSignatureBlock(Signature signature, X509Certificate publicKey, OutputStream out)
throws IOException, GeneralSecurityException {
SignerInfo signerInfo = new SignerInfo(new X500Name(publicKey.getIssuerX500Principal().getName()),
publicKey.getSerialNumber(), AlgorithmId.get("SHA1"), AlgorithmId.get("RSA"), signature.sign());
PKCS7 pkcs7 = new PKCS7(new AlgorithmId[] { AlgorithmId.get("SHA1") }, new ContentInfo(ContentInfo.DATA_OID,
null), new X509Certificate[] { publicKey }, new SignerInfo[] { signerInfo });
pkcs7.encodeSignedData(out);
}
private static void signWholeOutputFile(byte[] zipData, OutputStream outputStream, X509Certificate publicKey,
PrivateKey privateKey) throws IOException, GeneralSecurityException {
// For a zip with no archive comment, the
// end-of-central-directory record will be 22 bytes long, so
// we expect to find the EOCD marker 22 bytes from the end.
if (zipData[zipData.length - 22] != 0x50 || zipData[zipData.length - 21] != 0x4b
|| zipData[zipData.length - 20] != 0x05 || zipData[zipData.length - 19] != 0x06) {
throw new IllegalArgumentException("zip data already has an archive comment");
}
Signature signature = Signature.getInstance("SHA1withRSA");
signature.initSign(privateKey);
signature.update(zipData, 0, zipData.length - 2);
ByteArrayOutputStream temp = new ByteArrayOutputStream();
// put a readable message and a null char at the start of the
// archive comment, so that tools that display the comment
// (hopefully) show something sensible.
// TODO: anything more useful we can put in this message?
byte[] message = "signed by SignApk".getBytes("UTF-8");
temp.write(message);
temp.write(0);
writeSignatureBlock(signature, publicKey, temp);
int total_size = temp.size() + 6;
if (total_size > 0xffff) {
throw new IllegalArgumentException("signature is too big for ZIP file comment");
}
// signature starts this many bytes from the end of the file
int signature_start = total_size - message.length - 1;
temp.write(signature_start & 0xff);
temp.write((signature_start >> 8) & 0xff);
// Why the 0xff bytes? In a zip file with no archive comment,
// bytes [-6:-2] of the file are the little-endian offset from
// the start of the file to the central directory. So for the
// two high bytes to be 0xff 0xff, the archive would have to
// be nearly 4GB in side. So it's unlikely that a real
// commentless archive would have 0xffs here, and lets us tell
// an old signed archive from a new one.
temp.write(0xff);
temp.write(0xff);
temp.write(total_size & 0xff);
temp.write((total_size >> 8) & 0xff);
temp.flush();
// Signature verification checks that the EOCD header is the
// last such sequence in the file (to avoid minzip finding a
// fake EOCD appended after the signature in its scan). The
// odds of producing this sequence by chance are very low, but
// let's catch it here if it does.
byte[] b = temp.toByteArray();
for (int i = 0; i < b.length - 3; ++i) {
if (b[i] == 0x50 && b[i + 1] == 0x4b && b[i + 2] == 0x05 && b[i + 3] == 0x06) {
throw new IllegalArgumentException("found spurious EOCD header at " + i);
}
}
outputStream.write(zipData, 0, zipData.length - 2);
outputStream.write(total_size & 0xff);
outputStream.write((total_size >> 8) & 0xff);
temp.writeTo(outputStream);
}
private static PrivateKey readPrivateKey(File file)
throws IOException, GeneralSecurityException {
DataInputStream input = new DataInputStream(new FileInputStream(file));
try {
byte[] bytes = new byte[(int) file.length()];
input.read(bytes);
// dont support encrypted keys atm
//KeySpec spec = decryptPrivateKey(bytes, file);
//if (spec == null) {
KeySpec spec = new PKCS8EncodedKeySpec(bytes);
//}
try {
return KeyFactory.getInstance("RSA").generatePrivate(spec);
} catch (InvalidKeySpecException ex) {
return KeyFactory.getInstance("DSA").generatePrivate(spec);
}
} finally {
input.close();
}
}
private static X509Certificate readPublicKey(File file)
throws IOException, GeneralSecurityException {
FileInputStream input = new FileInputStream(file);
try {
CertificateFactory cf = CertificateFactory.getInstance("X.509");
return (X509Certificate) cf.generateCertificate(input);
} finally {
input.close();
}
}
public static void main(String[] args) {
if (args.length < 4) {
System.out.println("MinSignAPK pemfile pk8file inzip outzip");
System.out.println("- only adds whole-file signature to zip");
return;
}
String pemFile = args[0];
String pk8File = args[1];
String inFile = args[2];
String outFile = args[3];
try {
X509Certificate publicKey = readPublicKey(new File(pemFile));
PrivateKey privateKey = readPrivateKey(new File(pk8File));
InputStream fis = new FileInputStream(inFile);
byte[] buffer = new byte[(int)(new File(inFile)).length()];
fis.read(buffer);
fis.close();
OutputStream fos = new FileOutputStream(outFile, false);
signWholeOutputFile(buffer, fos, publicKey, privateKey);
fos.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}

BIN
server/minsignapk.jar Normal file

Binary file not shown.

153
server/opendelta.sh Normal file
View File

@ -0,0 +1,153 @@
#!/bin/bash
# Script to generate delta files for OpenDelta - by Jorrit 'Chainfire' Jongma
# Get device either from $DEVICE set by calling script, or first parameter
if [ "$DEVICE" == "" ]; then
if [ "$1" != "" ]; then
DEVICE=$1
fi
fi
if [ "$DEVICE" == "" ]; then
echo "Abort: no device set" >&2
exit 1
fi
# ------ CONFIGURATION ------
HOME=/home/build
BIN_JAVA=java
BIN_MINSIGNAPK=$HOME/delta/minsignapk.jar
BIN_XDELTA=$HOME/delta/xdelta3
BIN_ZIPADJUST=$HOME/delta/zipadjust
FILE_MATCH=omni-*.zip
PATH_CURRENT=$HOME/omni/out/target/product/$DEVICE
PATH_LAST=$HOME/delta/last/$DEVICE
KEY_X509=$HOME/.keys/platform.x509.pem
KEY_PK8=$HOME/.keys/platform.pk8
# ------ PROCESS ------
getFileName() {
echo ${1##*/}
}
getFileNameNoExt() {
echo ${1%.*}
}
getFileMD5() {
TEMP=$(md5sum -b $1)
for T in $TEMP; do echo $T; break; done
}
getFileSize() {
echo $(stat --print "%s" $1)
}
FILE_CURRENT=$(getFileName $(ls -1 $PATH_CURRENT/$FILE_MATCH))
FILE_LAST=$(getFileName $(ls -1 $PATH_LAST/$FILE_MATCH))
FILE_LAST_BASE=$(getFileNameNoExt $FILE_LAST)
if [ "$FILE_CURRENT" == "" ]; then
echo "Abort: CURRENT zip not found" >&2
exit 1
fi
if [ "$FILE_LAST" == "" ]; then
echo "Abort: LAST zip not found" >&2
mkdir -p $PATH_LAST
cp $PATH_CURRENT/$FILE_CURRENT $PATH_LAST/$FILE_CURRENT
exit 0
fi
if [ "$FILE_LAST" == "$FILE_CURRENT" ]; then
echo "Abort: CURRENT and LAST zip have the same name" >&2
exit 1
fi
rm -rf work
mkdir work
rm -rf out
mkdir out
$BIN_ZIPADJUST --decompress $PATH_CURRENT/$FILE_CURRENT work/current.zip
$BIN_ZIPADJUST --decompress $PATH_LAST/$FILE_LAST work/last.zip
$BIN_JAVA -Xmx1024m -jar $BIN_MINSIGNAPK $KEY_X509 $KEY_PK8 work/current.zip work/current_signed.zip
$BIN_JAVA -Xmx1024m -jar $BIN_MINSIGNAPK $KEY_X509 $KEY_PK8 work/last.zip work/last_signed.zip
$BIN_XDELTA -9evfS none -s work/last.zip work/current.zip out/$FILE_LAST_BASE.update
$BIN_XDELTA -9evfS none -s work/current.zip work/current_signed.zip out/$FILE_LAST_BASE.sign
MD5_CURRENT=$(getFileMD5 $PATH_CURRENT/$FILE_CURRENT)
MD5_CURRENT_STORE=$(getFileMD5 work/current.zip)
MD5_CURRENT_STORE_SIGNED=$(getFileMD5 work/current_signed.zip)
MD5_LAST=$(getFileMD5 $PATH_LAST/$FILE_LAST)
MD5_LAST_STORE=$(getFileMD5 work/last.zip)
MD5_LAST_STORE_SIGNED=$(getFileMD5 work/last_signed.zip)
MD5_UPDATE=$(getFileMD5 out/$FILE_LAST_BASE.update)
MD5_SIGN=$(getFileMD5 out/$FILE_LAST_BASE.sign)
SIZE_CURRENT=$(getFileSize $PATH_CURRENT/$FILE_CURRENT)
SIZE_CURRENT_STORE=$(getFileSize work/current.zip)
SIZE_CURRENT_STORE_SIGNED=$(getFileSize work/current_signed.zip)
SIZE_LAST=$(getFileSize $PATH_LAST/$FILE_LAST)
SIZE_LAST_STORE=$(getFileSize work/last.zip)
SIZE_LAST_STORE_SIGNED=$(getFileSize work/last_signed.zip)
SIZE_UPDATE=$(getFileSize out/$FILE_LAST_BASE.update)
SIZE_SIGN=$(getFileSize out/$FILE_LAST_BASE.sign)
DELTA=out/$FILE_LAST_BASE.delta
echo "{" > $DELTA
echo " \"version\": 1," >> $DELTA
echo " \"in\": {" >> $DELTA
echo " \"name\": \"$FILE_LAST\"," >> $DELTA
echo " \"size_store\": $SIZE_LAST_STORE," >> $DELTA
echo " \"size_store_signed\": $SIZE_LAST_STORE_SIGNED," >> $DELTA
echo " \"size_official\": $SIZE_LAST," >> $DELTA
echo " \"md5_store\": \"$MD5_LAST_STORE\"," >> $DELTA
echo " \"md5_store_signed\": \"$MD5_LAST_STORE_SIGNED\"," >> $DELTA
echo " \"md5_official\": \"$MD5_LAST\"" >> $DELTA
echo " }," >> $DELTA
echo " \"update\": {" >> $DELTA
echo " \"name\": \"$FILE_LAST_BASE.update\"," >> $DELTA
echo " \"size\": $SIZE_UPDATE," >> $DELTA
echo " \"size_applied\": $SIZE_CURRENT_STORE," >> $DELTA
echo " \"md5\": \"$MD5_UPDATE\"," >> $DELTA
echo " \"md5_applied\": \"$MD5_CURRENT_STORE\"" >> $DELTA
echo " }," >> $DELTA
echo " \"signature\": {" >> $DELTA
echo " \"name\": \"$FILE_LAST_BASE.sign\"," >> $DELTA
echo " \"size\": $SIZE_SIGN," >> $DELTA
echo " \"size_applied\": $SIZE_CURRENT_STORE_SIGNED," >> $DELTA
echo " \"md5\": \"$MD5_SIGN\"," >> $DELTA
echo " \"md5_applied\": \"$MD5_CURRENT_STORE_SIGNED\"" >> $DELTA
echo " }," >> $DELTA
echo " \"out\": {" >> $DELTA
echo " \"name\": \"$FILE_CURRENT\"," >> $DELTA
echo " \"size_store\": $SIZE_CURRENT_STORE," >> $DELTA
echo " \"size_store_signed\": $SIZE_CURRENT_STORE_SIGNED," >> $DELTA
echo " \"size_official\": $SIZE_CURRENT," >> $DELTA
echo " \"md5_store\": \"$MD5_CURRENT_STORE\"," >> $DELTA
echo " \"md5_store_signed\": \"$MD5_CURRENT_STORE_SIGNED\"," >> $DELTA
echo " \"md5_official\": \"$MD5_CURRENT\"" >> $DELTA
echo " }" >> $DELTA
echo "}" >> $DELTA
mkdir publish >/dev/null 2>/dev/null
mkdir publish/$DEVICE >/dev/null 2>/dev/null
cp out/* publish/$DEVICE/.
rm -rf work
rm -rf out
rm -rf $PATH_LAST/*
mkdir -p $PATH_LAST
cp $PATH_CURRENT/$FILE_CURRENT $PATH_LAST/$FILE_CURRENT
exit 0