Skip to content

Commit 546ed56

Browse files
raydacraydac
committed
Create README.md
1 parent 1df77e4 commit 546ed56

File tree

1 file changed

+268
-0
lines changed

1 file changed

+268
-0
lines changed

‎README.md‎

Lines changed: 268 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,268 @@
1+
Introduction
2+
=============
3+
Sometime it is very important to parse in Java some binary block data, may be not in very fast way but structures can have complex format and byte and bit orders can be very different. In Python we have [the Struct package](https://docs.python.org/2/library/struct.html) for operations to parse binary blocks but in Java such operations look a bit verbose and take some time to be programmed, so that I decided during my vacation to develop a framework which would decrease verbosity for such operations in Java and will decrease my work in future because I am too lazy to write a lot of code.<br>
4+
p.s.<br>
5+
For instance I have been very actively using the framework in [the ZX-Poly emulator](https://github.com/raydac/zxpoly) to parse snapshot files and save results.
6+
7+
Change log
8+
===========
9+
- 1.1.0
10+
- Added support for mapped classes output into JBBPOut
11+
- Added JBBPTextWriter to log binary data as text with commentaries,tabs and separators
12+
- Fixed read byte counter, now it counts only fully processed bytes, if only several bits have been read from byte then the byte will not be counted until whole read
13+
- Fixed static fields including in mapping processes if class has marked by default Bin annotation
14+
- Added flag JBBPParser#FLAG_SKIP_REMAINING_FIELDS_IF_EOF to ignore remaining fields during parsing if EOF without exception
15+
- Added flag JBBPMapper#FLAG_IGNORE_MISSING_VALUES to ignore mapping for values which are not found in parsed source
16+
- Added new auxiliary methods in JBBPUtils
17+
- 1.0
18+
- The Initial version
19+
20+
Java support
21+
=============
22+
The Framework developed to support Java platform since Java SE 1.5+, its inside mapping has been developed in such manner to support work under the Android platform (as of Android 2.0) so that the framework is an Android compatible one. It doesn't use any NIO and usage of the non-standard sun.misc.unsafe class is isolated with reflection.
23+
24+
How to use with Maven
25+
======================
26+
The Framework is published in the Maven Central thus it can be added as a dependency into a project
27+
```
28+
<dependency>
29+
<groupId>com.igormaznitsa</groupId>
30+
<artifactId>jbbp</artifactId>
31+
<version>1.1.0</version>
32+
</dependency>
33+
```
34+
also the precompiled jar, javadoc and sources can be downloaded manually from [the Maven central.](http://search.maven.org/#browse|808871750)
35+
36+
Hello world
37+
============
38+
The Framework is very easy in use because it has only two main classes for its functionality com.igormaznitsa.jbbp.JBBPParser (for data parsing) and com.igormaznitsa.jbbp.io.JBBPOut (for binary block writing), both of them work over low-level IO classes com.igormaznitsa.jbbp.io.JBBPBitInputStream and com.igormaznitsa.jbbp.io.JBBPBitOutputStream which are the core for the framework.
39+
```Java
40+
class Mapped { @Bin(type = BinType.BYTE_ARRAY) String text;}
41+
Mapped mapped = JBBPParser.prepare("byte [_] text;").parse(JBBPOut.BeginBin().Byte("Hello World").End().toByteArray()).mapTo(Mapped.class);
42+
assertEquals("Hello World",mapped.text);
43+
```
44+
More compex example with features added as of 1.1.0
45+
====================================================
46+
The Example shows how to parse a byte written in non-standard MSB0 order (Java has LSB0 bit order) to bit fields, print its values and pack fields back
47+
```Java
48+
class Flags {
49+
@Bin(outOrder = 1, name = "f1", type = BinType.BIT, outBitNumber = JBBPBitNumber.BITS_1, comment = "It's flag one") byte flag1;
50+
@Bin(outOrder = 2, name = "f2", type = BinType.BIT, outBitNumber = JBBPBitNumber.BITS_2, comment = "It's second flag") byte flag2;
51+
@Bin(outOrder = 3, name = "f3", type = BinType.BIT, outBitNumber = JBBPBitNumber.BITS_1, comment = "It's 3th flag") byte flag3;
52+
@Bin(outOrder = 4, name = "f4", type = BinType.BIT, outBitNumber = JBBPBitNumber.BITS_4, comment = "It's 4th flag") byte flag4;
53+
}
54+
55+
final int data = 0b10101010;
56+
Flags parsed = JBBPParser.prepare("bit:1 f1; bit:2 f2; bit:1 f3; bit:4 f4;", JBBPBitOrder.MSB0).parse(new byte[]{(byte)data}).mapTo(Flags.class);
57+
assertEquals(1,parsed.flag1);
58+
assertEquals(2,parsed.flag2);
59+
assertEquals(0,parsed.flag3);
60+
assertEquals(5,parsed.flag4);
61+
62+
System.out.println(new JBBPTextWriter().Bin(parsed).Close().toString());
63+
64+
assertEquals(data, JBBPOut.BeginBin(JBBPBitOrder.MSB0).Bin(parsed).End().toByteArray()[0] & 0xFF);
65+
```
66+
The Example will print in console the text below
67+
```
68+
;--------------------------------------------------------------------------------
69+
; START : Flags
70+
;--------------------------------------------------------------------------------
71+
01; f1, It's flag one
72+
02; f2, It's second flag
73+
00; f3, It's 3th flag
74+
05; f4, It's 4th flag
75+
;--------------------------------------------------------------------------------
76+
; END : Flags
77+
;--------------------------------------------------------------------------------
78+
```
79+
Format of a field
80+
==================
81+
Each field can be just a field or an array of fields, also it can be anonymous one or named one.
82+
83+
[<|>]field_type [[array_size|expression|_]] [field_name] ;
84+
85+
The first char shows the byte order to be used for parsing of the field (if the field is a multi-byte one) and can be omitted (and in the case the default order for the parser will be used). The Parser allows to parse a field as Big-Endinan one (for '>' prefix, it is the default state so that it can be omitted) and Little-Endian one (for '<' prefix). [You can read about endianness in wikipedia.](http://en.wikipedia.org/wiki/Endianness)
86+
87+
**The Field name will be normalized to its lower-case representation, so that keep in your mind that names are case insensitive.**
88+
89+
Supported data types and commands
90+
==================================
91+
- **bit[:<number Of bits>]** - a bit field of fixed size (1..7 bits), by default 1
92+
- **byte** - a signed byte field (8 bits)
93+
- **ubyte** - a unsigned byte field (8 bits)
94+
- **bool** - a boolean field (1 byte)
95+
- **short** - a signed short field (2 bytes)
96+
- **ushort**- a unsigned short field (2 bytes)
97+
- **int** - an integer field (4 bytes)
98+
- **long** - a long field (8 bytes)
99+
- **align[:<number Of bytes>]** - align the counter for number of bytes, by default 1. NB: It works relative to the current read byte counter!
100+
- **skip[:<number Of bytes>]** - skip number of bytes, by default 1
101+
- **var[:<numeric value>]** - a var field which should be read through an external processor defined by the user
102+
- **reset$$** - reset the input stream read byte counter, it is very useful for relative alignment operations
103+
104+
Structures
105+
===========
106+
Fields can be collected in structures, the format of structure definition:
107+
```
108+
[structure_name] [[array_size|expression|_]] { fields... }
109+
```
110+
Structures can contain another structures and every field inside a structure has path in the format ***structure_name.field_name*** but if you want use field names inside the same structure then use their names without name of structure.
111+
112+
Expressions
113+
============
114+
Expressions are used for calculation of length of arrays and allow brackets and integer operators which work similar to Java operators:
115+
- Arithmetic operators: +,-,%,*,/,%
116+
- Bit operators: &,|,^,~
117+
- Shift operators: <<,>>,>>>
118+
- Brackets: (, )
119+
120+
Inside expression you can use integer numbers and named field values through their names (if you use fields from the same structure) or paths. Keep in your mind that you can't use array fields or fields placed inside structure arrays.
121+
```
122+
int field1;
123+
struct1 {
124+
int field2;
125+
}
126+
byte [field1+struct1.field2] data;
127+
```
128+
129+
Field names
130+
============
131+
You can use any chars for field names but you can't use '.' in names (because the special char is used as the separator in field paths) and a name must not be started with a number. Also you must not use the symbols '$' and '_' as the first char in the name because they are used by the parser for special purposes.
132+
133+
Commentaries
134+
=============
135+
You can use commentaries inside a parser script, the parser supports the only comment format and recognizes as commentaries all text after '//' till the end of line.
136+
```
137+
int;
138+
// hello commentaries
139+
byte field;
140+
```
141+
142+
Expression macroses
143+
====================
144+
Inside expression you can use field names and field paths, also you can use the special macros '$$' which represents the current input stream byte counter, all fields started with '$' will be recognized by the parser as special user defined variables and it will be requesting them from special user defined provider. If the array size contains the only '_' symbol then the field or structure will not have defined size and whole stream will be read.
145+
146+
How to get result of parsing
147+
=============================
148+
The Result of parsing is an instance of com.igormaznitsa.jbbp.model.JBBPFieldStruct class which represents the root invisible structure for the parsed data and you can use its inside methods to find desired fields for their names, paths or classes. All Fields are successors of com.igormaznitsa.jbbp.model.JBBPAbstractField class. To increase comfort, it is easier to use mapping to classes when the mapper automaticaly places values to fields of a Java class.
149+
150+
Example
151+
========
152+
The Example below shows how to parse a PNG file with the JBBP parser (the example taken from tests)
153+
```Java
154+
final InputStream pngStream = getResourceAsInputStream("picture.png");
155+
try {
156+
157+
final JBBPParser pngParser = JBBPParser.prepare(
158+
"long header;"
159+
+ "// chunks\n"
160+
+ "chunk [_]{"
161+
+ " int length; "
162+
+ " int type; "
163+
+ " byte[length] data; "
164+
+ " int crc;"
165+
+ "}"
166+
);
167+
168+
final JBBPFieldStruct result = pngParser.parse(pngStream);
169+
170+
assertEquals(0x89504E470D0A1A0AL,result.findFieldForNameAndType("header",JBBPFieldLong.class).getAsLong());
171+
172+
final JBBPFieldArrayStruct chunks = result.findFieldForNameAndType("chunk", JBBPFieldArrayStruct.class);
173+
174+
175+
final String [] chunkNames = new String[]{"IHDR","gAMA","bKGD","pHYs","tIME","tEXt","IDAT","IEND"};
176+
final int [] chunkSizes = new int[]{0x0D, 0x04, 0x06, 0x09, 0x07, 0x19, 0x0E5F, 0x00};
177+
178+
assertEquals(chunkNames.length,chunks.size());
179+
180+
for(int i=0;i<chunks.size();i++){
181+
assertChunk(chunkNames[i], chunkSizes[i], (JBBPFieldStruct)chunks.getElementAt(i));
182+
}
183+
}
184+
finally {
185+
closeResource(pngStream);
186+
}
187+
```
188+
Also it is possible to map parsed packet to class fields
189+
```Java
190+
final JBBPParser pngParser = JBBPParser.prepare(
191+
"long header;"
192+
+ "chunk [_]{"
193+
+ " int length; "
194+
+ " int type; "
195+
+ " byte[length] data; "
196+
+ " int crc;"
197+
+ "}"
198+
);
199+
200+
class Chunk {
201+
@Bin int length;
202+
@Bin int type;
203+
@Bin byte [] data;
204+
@Bin int crc;
205+
}
206+
207+
@Bin
208+
class Png {
209+
long header;
210+
Chunk [] chunk;
211+
}
212+
213+
final Png png = pngParser.parse(pngStream).mapTo(Png.class);
214+
```
215+
The Example from tests shows how to parse a tcp frame wrapped in a network frame
216+
```Java
217+
final JBBPParser tcpParser = JBBPParser.prepare(
218+
"skip:34; // skip bytes till the frame\n"
219+
+ "ushort SourcePort;"
220+
+ "ushort DestinationPort;"
221+
+ "int SequenceNumber;"
222+
+ "int AcknowledgementNumber;"
223+
224+
+ "bit:1 NONCE;"
225+
+ "bit:3 RESERVED;"
226+
+ "bit:4 HLEN;"
227+
228+
+ "bit:1 FIN;"
229+
+ "bit:1 SYN;"
230+
+ "bit:1 RST;"
231+
+ "bit:1 PSH;"
232+
+ "bit:1 ACK;"
233+
+ "bit:1 URG;"
234+
+ "bit:1 ECNECHO;"
235+
+ "bit:1 CWR;"
236+
237+
+ "ushort WindowSize;"
238+
+ "ushort TCPCheckSum;"
239+
+ "ushort UrgentPointer;"
240+
+ "byte [$$-34-HLEN*4] Option;"
241+
+ "byte [_] Data;"
242+
);
243+
244+
final JBBPFieldStruct result = pngParser.parse(tcpFrameStream);
245+
```
246+
247+
My Binary data format is too complex one to be decoded by a JBBP script
248+
========================================================================
249+
No problems! The Parser works over com.igormaznitsa.jbbp.io.BitInputStream class which can be used directly and allows read bits, bytes, count bytes and align data from a stream.
250+
251+
I want to make a bin block instead of parsing!
252+
===============================================
253+
The Framework contains a special helper as the class com.igormaznitsa.jbbp.io.JBBPOut which allows to build bin blocks with some kind of DSL
254+
```Java
255+
import static com.igormaznitsa.jbbp.io.JBBPOut.*;
256+
...
257+
final byte [] array =
258+
BeginBin().
259+
Bit(1, 2, 3, 0).
260+
Bit(true, false, true).
261+
Align().
262+
Byte(5).
263+
Short(1, 2, 3, 4, 5).
264+
Bool(true, false, true, true).
265+
Int(0xABCDEF23, 0xCAFEBABE).
266+
Long(0x123456789ABCDEF1L, 0x212356239091AB32L).
267+
End().toByteArray();
268+
```

0 commit comments

Comments
 (0)