Subversion Repositories svnkaklik

\chap Trial design

The whole design of radioastronomy receiver digitalization unit is constructed to be used in a wide range of applications and tasks related to digitalization of signal from radioastronomy receivers. A good illustrating problem for its use is a signal digitalisation from multiple antenna arrays. This design will eventually become a part of MLAB Advanced Radio Astronomy System. 

\sec Required parameters

Wide dynamical range and high IP3 are desired. The receiver must accept wide dynamic signals because a typical radioastronomical signal has a form of a weak signal covered by a strong man-made noise.    

Summary of main required parameters follows 

\begitems

  * Dynamical range better than 80 dB

  * Phase stability between channels 

  * Noise (all types)

  * Sampling jitter better than 100 metres

  * Support for any number of receivers in range 1 to 8

\enditems

\sec Sampling frequency

Sampling frequency is limited by the technical constrains in the trial design. This parameter is especially limited by the sampling frequencies of analog-to-digital conversion chips available on the market and interface bandwidth. Combination of the required parameters -- dynamic range requiring at least 16bit and a minimum sampling frequency of 1 MSPS leads to need of high end ADC chips which does not support such low sampling frequencies at all. Their minimum sampling frequency is 5 MSPS.

We calculate minimum data bandwidth data rate for eight receivers, 2 bytes per sample and 5MSPS as $8 * 2 * 5e6 = 80$ MB/s. Such data rate is at the limit of real writing speed o classical HDD and it is almost double of real bandwidth of USB 2.0 interface. 

\sec System scalability

For analogue channels scalability, special parameters of ADC modules are required. Ideally, there should be a separate output for each analogue channel in ADC module. ADC module must also have separate outputs for frames and data output clocks. These parameters allow for conduction at relatively low digital data rates. As a result, the digital signal can be conducted even through long wires. 

Clock signal will be handled distinctively in our scalable design. Selected ADC chip are guaranteed to have defined clock skew between sampling and data output clock. This allows taking data and frame clocks from the first ADC module only. The rest of the data and frame clocks from other ADC modules can be measured for diagnostic purposes (failure detection, jitter measurement etc.).

This system concept allows for scalability, that is technically limited by a number of differential signals on host side and its computational power.  There is another advantage of scalable data acquisition system -- an economic one. Observatories or end users can make a choice of how much money are they willing to spent on radioastronomy receiver system. This freedom of choice is especially useful for science sites without previous experience in radioastronomy observations.     

\secc Differential signaling 

The concept of scalable design requires relatively long circuit traces between ADC and digital unit which captures the data and performs the computations. The long distance between the digital processing unit and the analog-to-digital conversion unit has an advantage in noise retention typically produced by digital circuits. Those digital circuits, such as FPGA or other flip-flops block and circuit traces, usually work at high frequencies and emit wide-band noise with relatively low power. In such cases any increase in a distance between the noise source and analog signal source increase S/N significantly. However, at the same time a long distance brings problems with the digital signal transmission between ADC and computational unit. This obstacle should be resolved more easily in free-space than on board routing. The high-quality differential signalling shielded cables should be used. This technology has two advantages over PCB signal routing. First, it can use twisted pair of wires for leak inductance suppression in signal path and second, the twisted pair may additionally be shielded by uninterrupted metal foil.              

\secc Phase matching

For multiple antenna radioastronomy projects, system phase stability is a mandatory condition. It allows precise high resolution imaging of objects. 

High phase stability in our scalable design is achieved through centralized frequency generation  and distribution with multi-output LVPECL hubs, that have equiphased outputs for multiple devices. 

This design ensures that all devices have access to the defined phase and known frequency.     

\sec System description

In this section testing system will be described.

\secc Frequency synthesis       

We have used a centralized topology as a basis for frequency synthesis. One precise high-frequency and low-jitter digital oscillator has been used \cite[MLAB-GPSDO], while other working frequencies have been derived from it by the division of its signal. This central oscillator has a software defined GPS disciplined control loop for frequency stabilization.\fnote{SDGPSDO design has been developed in parallel to this diploma thesis as a related project, but it is not explicitly required by the diploma thesis.}

We have used methods of frequency monitoring compensation in order to meet modern requirements on radioastronomy equipment which needs precise frequency and phase stability over a wide scale for effective radioastronomy imaging. 

GPSDO device consists the Si570 chip with LVPECL output. Phase jitter of GPSDO is determined mainly by Si570 phase noise. Parameters of used Si570 from source ... are summarised in table.

		dBc/Hz		

Offset Frequency		156.25 MHz	&	622.08 MHz

100 Hz	&	–105	&	–97

1 kHz	&	–122	&	–107

10 kHz	&	–128	&	–116

100 kHz	&	–135	&	–121

1 MHz	&	–144	&	–134

10 MHz	&	–147	&	–146

100 MHz	&	n/a	&	–148

Every ADC module will be directly connected to CLKHUB02A module which takes sampling clock signal delivered by FPGA from main local oscillator.  This signal should use high quality differential signaling cable -- we should use SATA cable for this purpose. 

GPSDO design included in data acquisition system has special feature -- generates time marks for precise time-stamping of received signal. Timestamps are created by disabling of local oscillator for 100 us as result rectangle click in input signal is created which appears as horizontal line in spectrogram.   

Timestamps should be seen in image \ref[meteor-reflection] (above and below meteor reflection).

Time-marking should be improved in future by digitalisation GPS signal directly with dedicated ADC channel.  Datafile then consists samples from channels of radio-astronomy receivers along with GPS signal containing precise time information. 

\secc Signal cable connectors 

Several widely used and commercially easily accessible differential connectors were considered to be use in our design. 

\begitems

* HDMI % [[http://en.wikipedia.org/wiki/Hdmi|HDMI]]</del>

* SATA  		%{http://en.wikipedia.org/wiki/Serial_attached_SCSI#Connectors|SAS]]/[[http://en.wikipedia.org/wiki/Serial_ATA|SATA]]

* DisplayPort 		%[[http://en.wikipedia.org/wiki/Display_port|DisplayPort]]</del>

* SAS/miniSAS

\enditems

At the end, MiniSAS connector was chosen as the best option to be used in connecting together multiple ADC modules. It is compatible with existing SATA cabling systems and aggregates multiple SATA cables to a single connector. It can be seen on the following picture \ref[img-miniSAS-cable]. A transition between SATA and miniSAS is achieved by SAS to SATA adapter cable which is commonly used in servers to connect SAS controller to multiple SATA hard disc in RAID systems and thus is commercially easily available. 

The main drawback of miniSAS PCB connectors lies in the fact, that they are manufactured in SMT versions only. The outer metal housing of connector is designed to be mounted using a standard through-hole mounting scheme, a design that unfortunately decreases the durability of the connector. 

\midinsert

\clabel[img-miniSAS-cable]{Used miniSAS cable}

\picw=10cm \cinspic ./img/miniSAS_SATA_cable.jpg

\caption/f A type of miniSAS cable similar to used.

\endinsert

\secc Signal integrity requirements

\label[diff-signaling]

We use ADC modules that have DATA clock frequency eight times higher than sampling frequency in single line output mode, implying a 40 MHz output bit rate. This imply $ 1/4*10^7 = 25$ ns time length of data bit, which is equivalent to 7.5m light path in free space. If we use copper PCB with FR4 substrate layer or coaxial/twinax cable, we could obtain velocity factor of 0.66 at worst condition. Then the light path for the same bit rate will be 4.95 m. Although we do not have any cables in system with comparable lengths, worst data bit skew described by data sheets of used components is $0.3*sample time$ time which is 1.485 m. Therefore length matching is not critical in our design. 

\secc ADC modules design

\secc ADC selection

There exist several ADC signaling formats currently used in communication with FPGA. 

\begitems

  * DDR LVDS

  * JEDEC 204B

  * JESD204A

  * Paralel LVDS

  * Serdes

  * serial LVDS

\enditems

Because it uses the smallest number of differential pairs, the choice fell on the serial LVDS format. Small number of differential pairs is an important parameter determining the construction complexity and reliability\cite[serial-lvds]. 

An ultrasound AFE chip seems to be ideal for this purpose -- the chip has integrated both front-end amplifiers and filters. It has a drawback though - it is incapable of handling differential input signal and has a relatively low dynamic range (as it consists only of 12bit ADC). Because this IO has many ADC channels the scaling is possible only by a factor of 4 receivers (making 8 analogue channels).

If we require a separate output for every analogue channel and a 16bit depth we find that there are only a few 2-Channel simultaneous sampling ADCs currently existing which meet these requirements.  We have summarized the ADCs in the following table \ref[ADC-type] 

\midinsert \clabel[ADC-types]{Available ADC types}

\ctable{lrrrrrcc}{

\hfil ADC Type & LTC2271 & LTC2190 & LTC2191 & LTC2192 & LTC2193 & LTC2194 & LTC2195 \cr

SNR [dB] & 84.1 & 77 & 77 & 77 & 76.8 & 76.8 & 76.8 \cr

SFDR [dB] & 99 & 90 & 90 & 90 & 90 & 90 & 90 \cr

S/H Bandwidth [MHz] & 200 & \multispan6 550 \cr

Sampling rate [MSPS] & 20 & 25 & 40 & 65 & 80 &  105 & 125 \cr

Configuration & \multispan7 SPI \cr

Package & \multispan7 52-Lead (7mm $×$ 8mm) QFN \cr

}

\caption/t The summary of available ADC types and theirs characteristics. 

\endinsert

All parts in this category are compatible with one board layout. Main differences lay in the sampling frequency and signal to noise ratio, with the slowest having a maximum sampling frequency of 20 MHz. However all of them have a minimal sampling frequency of 5 MSPS and all are configurable over a serial interface (SPI). SPI seems to be a standard interface used in high-end ADC chips made by the largest manufacturers (Analog Devices, Linear technology, Texas instruments, Maxim integrated..). 

The ADC modules have a standard MLAB construction scheme with four mounting holes in corners aligned in defined raster. 

Data serial data outputs of ADC modules should be connected directly to FPGAs for the basic primary signal processing. The ADC chip used in the modules has a selectable bit width of data output bus and thus the output SATA connectors have signals arranged to contain a single bit from every ADC channel.  This creates a signal concept enabling a selection of a proper bus bit-width according to the sampling rate (higher bus bit-width downgrades signalling speed and vice versa.)

In order to connect the above mentioned signalling layout, miniSAS to multiple SATA cable should be used.  

A KiCAD design suite had been chosen for PCB layout. However, the version is, despite having integrated CERN Push \& Shove routing capability, slightly unstable as it sometimes crushes due to an exception during routing. On the basis of these stability issues, the design had to be saved quite often. On the other hand, compared to commercially available solutions, such as MentorGraphics PADS or Cadence Orcad,  the Open-source KiCAD provides an acceptable option and it easily surpasses a widely used Eagle software.

As a part of work on the thesis, new PCB footprints for FMC, SATA a and miniSAS connectors have been designed and were committed to KiCAD github library repository. They are now publicly available on the official KiCAD repository at GitHub.  

ADCdual01A module has several digital data output formats

\begitems

    * 1-lane mode

\enditems

All of these modes are supported by module design. For discused data acquisition system the 1-lane mode was selected. 1-lane mode allows minimal number of differential pairs between ADCdual01A and FPGA. Digital signaling scheme used in 1-lane mode is shown in image \ref[1-line-out]. 

\midinsert

\clabel[1-line-out]{Single line ADC output signals}

\picw=15cm \cinspic ./img/ADC_single_line_output.png

\caption/f Digital signaling shema for  1-line  ADC digital output mode.

\endinsert

ADCdual01A parameters can be set either by jumper setup (refered as parallel programming  in device's data sheet) or by SPI interface. SPI interface has been selected for our system, because parallel programming lacks of options (test pattern output setup for example). 

Complete schematic diagram of ADCdual01A module board is included in the appendix. 

\secc ADC modules interface

Both of the ADCdual01A modules were connected to FPGA ML605 board trough FMC2DIFF01A adapter board. The design of this adapter module expects the presence of FMC LPC connector and the board is, at the same time, not compatible with MLAB. It is, on the other hand, designed to meet the VITA 57 standard specifications for boards which support region 1 and region 3. VITA 57 regions are explained in the picture \ref[VITA57-regions].

This industry standard guarantees the compatibility with other FPGA boards that have FMC LPC connectors for Mezzanine Card. Schematic diagram of designed adapter board is included in the appendix. 

The primary purpose of the PCB is to enable the connection of ADC modules located outside the PC case. (In PC box analog circuits cannot be realised without the use of massive RFI mitigation techniques). 

Differential signalling connectors should be used for conducting digital signal over relatively long cables. The signal integrity sensitive links (clocks) are equipped with output driver and translator to LVPECL logic for better signal transmission quality.  

\midinsert

\picw=10cm \cinspic ./img/ML605-board.jpg

\caption/f FPGA ML605 development board.

\endinsert

\midinsert

\clabel[VITA57-regions]{VITA57 board geometry}

\picw=10cm \cinspic ./img/VITA57_regions.png

\caption/f Definition of VITA57 regions.

\endinsert

Several SATA connectors and two miniSAS connectors are populated on this board.  This set of connectors allows a connection of any number of ADC modules within the range of 1 to 8. ADC data outputs should be connected to the miniSAS connectors, while other supporting signals should be routed directly to SATA connectors on adapter. 

Lengths of differential pairs routed on PCB of module are not matched between pairs. Length variation of differential pairs is not critical in our design according to facts discussed in paragraph \ref[diff-signaling]. Nevertheless signals within differential pairs itself are matched for length. Internal signal traces length mating of differential pairs is mandatory in order to avoid dynamic logic hazard conditions on digital signals. Thus clocks signals are routed most precisely on all designed boards.

Signal configuration used in our trial design is described in the following tables. 

\secc Output data format

\midinsert

\ctable {clllllllll}{

\hfil

 & \multispan9 \hfil 160bit packet \hfil \crl \tskip4pt

Data name &  FRAME  & \multispan2 \hfil ADC1 CH1 \hfil & \multispan2 \hfil ADC1 CH2 \hfil & \multispan2  \hfil ADC2 CH1 \hfil & \multispan2 \hfil ADC2 CH2 \hfil  \cr

Data type & uint32 & int16 & int16 & int16 & int16 & int16 & int16 & int16 & int16 \cr

Content & saw signal & $t1$ &  $t_{1+1}$ &  $t1$ &  $t_{1+1}$ &  $t1$ &  $t_{1+1}$ &  $t1$ &  $t_{1+1}$ \cr

}

\caption/t System device "/dev/xillybus_data2_r" data format

\endinsert

\sec Achieved parameters

\secc Data reading and recording 

We use Gnuradio software to read the data stream from the ADC drive. Gnuradio suite consist of gnuradio-companion which is a graphical tool for creating signal-flow graphs and generating flow-graph source code. This tool was used to create a basic RAW data grabber to record and interactively view the data stream output from ADC modules. 

\midinsert

\picw=15cm \cinspic ./img/screenshots/Grabber.grc.png

\caption/f An ADC recorder flow graph created in gnuradio-companion.

\endinsert

\midinsert

\picw=15cm \cinspic ./img/screenshots/Grabber_running.png

\caption/f User interface window of a running ADC grabber.

\endinsert

Interactive grabber viewer user interface shows live oscilloscope-like time-value display for all data channels and live time-frequency scrolling display (a waterfall view) for displaying the frequency components of the grabbed signal. 

\secc ADC module parameters

Two pieces of ADC modules were completed and tested. The first piece, labeled ADC1, has LTC2190

ADC chip populated with LT6600-5 front-end operational amplifier. It also has a 1kOhm resistors populated on inputs which gives it an ability of an internal attenuation of input signal. The value of this attenuation $A$ is described by the following formula \ref[ADC1-gain]

\label[ADC1-gain]

$$

A = {806 \times R_1 \over R_1 + R_2}

$$

Where is 

\begitems

  * $A$ -  Gain of input amplifier.

  * $R_1$ - Output impedance of signal source (usually 50 Ohm).

  * $R_2$ - Value of serial resistors at operational amplifier inputs.

\enditems

We have $R_2 = 1000 \Omega$ and $R_1 = 50 \Omega$ which imply $A = 0.815$. That value of A is confirmed by measurement. 

In our measurement setup we have H1012 Ethernet transformer connected at inputs of ADC. Transformer has 10\% tolerance in impedance and amplification. We measured ADC saturation voltage 705.7 mV (generator output) in this setup due to impedance mismatch and uncalibrated transformer gain. 

\midinsert

\clabel[ADC1-FFT]{ADC1 sine test FFT}

\picw=15cm \cinspic ./img/screenshots/ADC1_CH2_FFT.png

\caption/f Sine signal sampled by ADC1 module with LTC2190 and LT6600-5 devices.

\endinsert

For ADC2 we must use formula with different constant \ref[ADC1-gain]. ADC2 module has LT6600-2.5 populated and gain is $A = 2.457$ with same $R_2$ resistors. We measured saturation voltage of 380 mV (generator output) at channel 1 on this ADC. It is well in parameter tolerances  of used setup.   

\label[ADC2-gain]

$$

A = {1580 \times R_1 \over R_1 + R_2}

$$

Where is 

\begitems

  * $A$ -  Gain of input amplifier.

  * $R_1$ - Output impedance of signal source (usually 50 Ohm).

  * $R_2$ - Value of serial resistors at operational amplifier inputs.

\enditems

\midinsert

\clabel[ADC2-FFT]{ADC2 sine test FFT}

\picw=15cm \cinspic ./img/screenshots/ADC2_CH1_FFT.png

\caption/f Sine signal sampled by ADC2 module with LTC2271 and LT6600-2.5 devices.

\endinsert

Computed FFT spectra for measured signal are shown in images \ref[ADC2-FFT] and \ref[ADC1-FFT].  Both images confirms that ADCdual01A modules have input dynamical range 80 dB at least. 

\chap Example of usage

For additional validation of system design a receiver setup was constructed. 

\sec Basic interferometer station

Interferometry station was selected as most basic setup. We connected the new data acquisition system to two SDRX01B receivers. Block schematic of used setup is shown in image \ref[block-schematic]. Two ground-plane antennas were used and mounted outside of balcony at CTU building at location 50$^\circ$4'36.102"N, 14 $^\circ$ 25'4.170" E. Antennas were equipped  by LNA01A amplifiers. Coaxial cable length are matched for 5 meters. And antennas were isolated by common mode ferrite bead mounted on cable for minimize signal coupling between antennas. Evaluation system consists SDGPSDO local oscillator subsystem used for tuning local oscillator frequency. 

\midinsert

\clabel[block-schematic]{Receiver block schematic}

\picw=10cm \cinspic ./img/Coherent_UHF_SDR_receiver.png

\caption/f Complete receiver block schematic of dual antenna interferometric station.

\endinsert

\midinsert

\clabel[meteor-reflection]{Meteor reflection}

\picw=10cm \cinspic ./img/screenshots/observed_meteor.png

\caption/f Meteor reflection received by evaluation setup.

\endinsert

\midinsert

\clabel[phase-phase-difference]{Phase difference}

\picw=10cm \cinspic ./img/screenshots/phase_difference.png

\caption/f Demonstration of phase difference between antennas.

\endinsert

%\sec Simple passive Doppler radar

%\sec Simple polarimeter station

\chap Proposed final system

Construction of a final system which is supposed to be employed for real radioastronomy observations will be described. This chapter is mainly a theoretical analysis of data handling systems. Realization of these ideas might be possible as a part of our future development after we fully evaluate and test the current trial design. 

The system requires proper handling of huge amounts of data and either huge and fast storage capacity is needed for store captured signal data, or enormous computational power is required for online data processing and filtering. Several hardware approach currently exist and are in use for data processing problem handling. Either powerful multi gigahertz CPUs, GPUs, FPGAs, or specially  constructed ASICs are used for this task.

\sec Custom design of FPGA board

In the beginning of the project, a custom design of FPGA interface board had been considered. This FPGA board should include PCI express interface and should sell at lower price than trial design. It should be compatible with MLAB which is further backward compatible with the existing or improved design of ADC modules. For a connection of this board to another adapter board with PCIe we expect a use of a host interface. 

Thunderbolt technology standard was expected to be used in this PC to PCIe -> FPGA module. Thunderbolt chips are currently available on the market for reasonable prices. However, a problem lies in the accessibility to their specifications, as they are only available for licensed users and Intel has a mass market oriented licensing policy, that makes this technology inaccessible for low quantity production. As a consequence, an external PCI Express cabling and expansion slots should be considered as a better solution. 

However, these systems and cables are still very expensive. Take Opal Kelly XEM6110 \cite[fpga-pcie] as an example, with its price tag reaching 995 USD at time of writing of thesis.

Therefore, a better solution probably needs to be found.

\sec Parralella board computer

%Parallella is gon

\sec GPU based computational system 

A new GPU development board NVIDIA K1, shown in the following picture \ref[img-NVIDIA-K1], has recently been released. These boards are intended to be used in fields including computer vision, robotics, medicine, security or automotive industry. They have ideal parameters for signal processing for a relatively low price of 192 USD.  Unfortunately, they are currently only in pre-order release stage (in April 2014). 

\midinsert

\clabel[img-NVIDIA-K1]{NVIDIA Jetson TK1 Development Kit}

\picw=15cm \cinspic ./img/Jetson_TK1_575px.jpg

\caption/f The NVIDIA Jetson TK1 Development Kit \url{https://developer.nvidia.com/jetson-tk1}.

\endinsert