forked from oracle/nvm-direct
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathREADME.html
387 lines (331 loc) · 14.8 KB
/
README.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
<!DOCTYPE html>
<html>
<head>
<title>README</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<style type="text/css">
/* GitHub stylesheet for MarkdownPad (http://markdownpad.com) */
/* Author: Nicolas Hery - http://nicolashery.com */
/* Version: b13fe65ca28d2e568c6ed5d7f06581183df8f2ff */
/* Source: https://github.com/nicolahery/markdownpad-github */
/* RESET
=============================================================================*/
html, body, div, span, applet, object, iframe, h1, h2, h3, h4, h5, h6, p, blockquote, pre, a, abbr, acronym, address, big, cite, code, del, dfn, em, img, ins, kbd, q, s, samp, small, strike, strong, sub, sup, tt, var, b, u, i, center, dl, dt, dd, ol, ul, li, fieldset, form, label, legend, table, caption, tbody, tfoot, thead, tr, th, td, article, aside, canvas, details, embed, figure, figcaption, footer, header, hgroup, menu, nav, output, ruby, section, summary, time, mark, audio, video {
margin: 0;
padding: 0;
border: 0;
}
/* BODY
=============================================================================*/
body {
font-family: Helvetica, arial, freesans, clean, sans-serif;
font-size: 14px;
line-height: 1.6;
color: #333;
background-color: #fff;
padding: 20px;
max-width: 960px;
margin: 0 auto;
}
body>*:first-child {
margin-top: 0 !important;
}
body>*:last-child {
margin-bottom: 0 !important;
}
/* BLOCKS
=============================================================================*/
p, blockquote, ul, ol, dl, table, pre {
margin: 15px 0;
}
/* HEADERS
=============================================================================*/
h1, h2, h3, h4, h5, h6 {
margin: 20px 0 10px;
padding: 0;
font-weight: bold;
-webkit-font-smoothing: antialiased;
}
h1 tt, h1 code, h2 tt, h2 code, h3 tt, h3 code, h4 tt, h4 code, h5 tt, h5 code, h6 tt, h6 code {
font-size: inherit;
}
h1 {
font-size: 28px;
color: #000;
}
h2 {
font-size: 24px;
border-bottom: 1px solid #ccc;
color: #000;
}
h3 {
font-size: 18px;
}
h4 {
font-size: 16px;
}
h5 {
font-size: 14px;
}
h6 {
color: #777;
font-size: 14px;
}
body>h2:first-child, body>h1:first-child, body>h1:first-child+h2, body>h3:first-child, body>h4:first-child, body>h5:first-child, body>h6:first-child {
margin-top: 0;
padding-top: 0;
}
a:first-child h1, a:first-child h2, a:first-child h3, a:first-child h4, a:first-child h5, a:first-child h6 {
margin-top: 0;
padding-top: 0;
}
h1+p, h2+p, h3+p, h4+p, h5+p, h6+p {
margin-top: 10px;
}
/* LINKS
=============================================================================*/
a {
color: #4183C4;
text-decoration: none;
}
a:hover {
text-decoration: underline;
}
/* LISTS
=============================================================================*/
ul, ol {
padding-left: 30px;
}
ul li > :first-child,
ol li > :first-child,
ul li ul:first-of-type,
ol li ol:first-of-type,
ul li ol:first-of-type,
ol li ul:first-of-type {
margin-top: 0px;
}
ul ul, ul ol, ol ol, ol ul {
margin-bottom: 0;
}
dl {
padding: 0;
}
dl dt {
font-size: 14px;
font-weight: bold;
font-style: italic;
padding: 0;
margin: 15px 0 5px;
}
dl dt:first-child {
padding: 0;
}
dl dt>:first-child {
margin-top: 0px;
}
dl dt>:last-child {
margin-bottom: 0px;
}
dl dd {
margin: 0 0 15px;
padding: 0 15px;
}
dl dd>:first-child {
margin-top: 0px;
}
dl dd>:last-child {
margin-bottom: 0px;
}
/* CODE
=============================================================================*/
pre, code, tt {
font-size: 12px;
font-family: Consolas, "Liberation Mono", Courier, monospace;
}
code, tt {
margin: 0 0px;
padding: 0px 0px;
white-space: nowrap;
border: 1px solid #eaeaea;
background-color: #f8f8f8;
border-radius: 3px;
}
pre>code {
margin: 0;
padding: 0;
white-space: pre;
border: none;
background: transparent;
}
pre {
background-color: #f8f8f8;
border: 1px solid #ccc;
font-size: 13px;
line-height: 19px;
overflow: auto;
padding: 6px 10px;
border-radius: 3px;
}
pre code, pre tt {
background-color: transparent;
border: none;
}
kbd {
-moz-border-bottom-colors: none;
-moz-border-left-colors: none;
-moz-border-right-colors: none;
-moz-border-top-colors: none;
background-color: #DDDDDD;
background-image: linear-gradient(#F1F1F1, #DDDDDD);
background-repeat: repeat-x;
border-color: #DDDDDD #CCCCCC #CCCCCC #DDDDDD;
border-image: none;
border-radius: 2px 2px 2px 2px;
border-style: solid;
border-width: 1px;
font-family: "Helvetica Neue",Helvetica,Arial,sans-serif;
line-height: 10px;
padding: 1px 4px;
}
/* QUOTES
=============================================================================*/
blockquote {
border-left: 4px solid #DDD;
padding: 0 15px;
color: #777;
}
blockquote>:first-child {
margin-top: 0px;
}
blockquote>:last-child {
margin-bottom: 0px;
}
/* HORIZONTAL RULES
=============================================================================*/
hr {
clear: both;
margin: 15px 0;
height: 0px;
overflow: hidden;
border: none;
background: transparent;
border-bottom: 4px solid #ddd;
padding: 0;
}
/* TABLES
=============================================================================*/
table th {
font-weight: bold;
}
table th, table td {
border: 1px solid #ccc;
padding: 6px 13px;
}
table tr {
border-top: 1px solid #ccc;
background-color: #fff;
}
table tr:nth-child(2n) {
background-color: #f8f8f8;
}
/* IMAGES
=============================================================================*/
img {
max-width: 100%
}
</style>
</head>
<body>
<h1>NVM Direct</h1>
<h2>Version 0.3</h2>
<p>The advent of persistent memory in enterprise computing introduces new
problems related to managing consistent application state. To solve
these problems Oracle is developing NVM Direct, an open source C library and a set of C language extensions
for applications utilizing NVM. This library and the extensions
implement an application interface (API) that is applicable to a wide
range of applications.</p>
<p>This repository contains the shared library portion of the API and an example application. It is mostly complete, but there are a few pieces missing. It has not been rigorously tested yet. This preliminary version is being released so people can understand the intent of the API, and get a feel for what it can do. This is not production quality.</p>
<p>There will be a GitHub repository containing the C extensions precompiler source
code when it is completed. For now the Linux executables for the precompiler can be
downloaded from the Oracle Technology Network at the <a href="http://www.oracle.com/technetwork/oracle-labs/open-nvm-download-2440119.html" title="NVM Direct">NVM Direct</a> page. The precompiler consists of two executables: clang and usidmap. The precompiler itself is a modified version of clang which takes the preprocessed version of code using the C extensions, and produces C source which gcc can compile. The precompiler also produces a JSON file describing all the persistent structs and callback functions. The JSON file is processed by usidmap to generate the code to register all the USID's for an application. The makefiles to use these executables are included in the eclipse DebugX build configurations for all projects. Extract the two executables and ensure they are available through your $PATH as clang and usidmap. This should work on any modern x64 Linux distribution.</p>
<p>The library is coded both with and without the
C extensions. If <code>NVM_EXT</code> is defined then the version with extensions
will be compiled. This will be set if the DebugX build configuration is used. The two versions provide a comparison of how the extensions will look when used in a real application. </p>
<p>The library has been developed using Eclipse Luna. There are three Eclipse projects in the repository for the NVM Direct code. There are two build configurations for each project. The configurations ending in X compile the code with the C
extensions. Of course that will not work without the precompiler.</p>
<p>There are two Eclipse projects for an example application. They are written with the C extensions and require the precompiler.</p>
<p>The bin project just contains a shell script to compile a C file that uses the NVM Direct C extensions. It is only intended to work with the makefiles generated by Eclipse Luna for gcc. However it should be easy to modify for other environments.</p>
<p>Note that most of the DebugX build configurations use a custom shell script to run usidmap and to post process its output. This is necessary because of some problems with usidmap. A future version will not require the post processing. The test_dmf project does not run usidmap because it does not define any USID's.</p>
<h3>Eclipse Projects:</h3>
<ul>
<li><strong><code>test_nvmd</code>:</strong> This is a silly NVM application that tries to get most
of the library code called. It forks itself and kills its child to
exercise recovery.</li>
<li><strong><code>nvmd</code>:</strong> This is the portable shared library. It contains the bulk of
the code.</li>
<li><strong><code>nvmds</code>:</strong> This is a service layer that allows the library to be customized for different environments. The current service layer is for testing the library on Linux. It does not actually work with persistent memory. It needs to integrate with Intel's libpmem to actually work with NVM.</li>
<li><strong><code>DirectMappedFile</code>:</strong> This is an example shared library written with the NVM Direct API. It uses NVM Direct to create a region file that supports atomic
block I/O to NVM using memcpy. It supports an arbitrary number of threads
simultaneously doing reads and writes. The size of the file can be changed
concurrently with I/O. This is similar to the functionality provided by <a href="http://pmem.io/2014/09/23/btt.html">Block Translation Table</a>. See dmf.h and dmf.c for more information.</li>
<li><strong><code>test_dmf</code>:</strong> This is an application to test DirectMappedFile. It gets multiple threads simultaneously reading/writing/resizing a file.</li>
<li><strong><code>bin</code>:</strong> This just contains a shell script for compiling sources that use the C extensions for NVM Direct.</li>
</ul>
<h3>Documents:</h3>
<p>There is some documentation in the root of the repository. This is intended
to aid in understanding the code.</p>
<ul>
<li><strong><code>LICENSE</code>:</strong> This contains the Universal Permissive License that
this code is released under.</li>
<li><strong><code>NVM_Direct_API.pdf</code>:</strong> This is an 100 page document describing the API and
the reasoning behind the design.
It includes the extended C syntax and all the publicly available library
entry points. It is not as detailed as man pages, but it is a start. </li>
<li><strong><code>NVM_Direct_Presentation.pdf</code>:</strong> This is a slide deck that has been used for presentations on the API. It is pretty basic.</li>
<li><strong><code>NVM_Direct_Notes.pdf</code>:</strong> This is the same presentation, but with
the notes for each slide. The notes give information which could not be
presented due to time constraints.</li>
</ul>
<h3>To Do:</h3>
<p>The API is not yet an industrial strength software package ready for use in production code. This alpha release has all the software interfaces that a production quality release will have, but lacks some of the external support that is really required. Some of the algorithms can also be improved.</p>
<ul>
<li>
<p>The error reporting is inadequate. Real developers will need much more information about asserts caused by improper library calls and NVM corruptions found by the library. An error reporting system needs to be developed.</p>
</li>
<li>
<p>The documentation does not go down to the level of detail a developer expects in a man page. The .h files are the best documentation. The man pages need to be written.</p>
</li>
<li>
<p>There are Doxygen annotated comments throughout the code, but there is no overall Doxygen documentation. Doxygen documentation needs to be generated and made complete.</p>
</li>
<li>
<p>There is a small test program used for initial debugging. However it is far from an adequate test of the entire API in all its corner cases. A full test suite needs to be developed.</p>
</li>
<li>
<p>The code contains a few TODO comments that create Eclipse tasks. These still need to be addressed. They do not significantly impact functionality.</p>
</li>
<li>
<p>The library does not actually work with persistent memory. The service layer uses a disk file system and does not actually do the necessary processor cache flushing and persistent memory barriers needed for real NVM. The service layer needs to be integrated with libpmem.</p>
</li>
<li>
<p>The precompiler that implements the language extensions is not complete yet, and will not be open sourced until it is in a better state. The library cannot be compiled with <code>NVM_EXT</code> defined without the precompiler. Development versions of the precompiler have successfully compiled the code with extensions, but not all of the extended features are available. The most significant issue is the automatic construction of the initialized nvm_type structs that describe the persistent structs. They have been hand crafted in usidmap.c. It is difficult to keep the C declarations and usidmap.c in sync.</p>
</li>
<li>
<p>Debugger support is needed for the language extensions. Following self-relative pointers in gdb is cumbersome at best. The debugger should be able to recognize USID's and report structs properly formatted. Getting formatted dumps of transaction undo could be very useful for resolving recovery problems.</p>
</li>
<li>
<p>Eclipse and other IDE's contain C parsers that do not recognize the NVM extensions for C. This makes some Eclipse features unusable with code that contains extensions. </p>
</li>
<li>
<p>There are some configuration parameters that the service layer passes to the generic layer of the library. Currently these are hard wired as numeric values in the service layer code. There needs to be a mechanism to allow per application configuration of the parameters.</p>
</li>
<li>
<p>Since the code has never been run on real NVM there are no performance numbers. Benchmarks need to be developed and run on real hardware to evaluate the library.</p>
</li>
<li>
<p>The code has not been instrumented to report performance statistics. This is needed for application tuning, and bottleneck discovery.</p>
</li>
</ul>
</body>
</html>
<!-- This document was created with MarkdownPad, the Markdown editor for Windows (http://markdownpad.com) -->