IBM Research - Australia

With the Internet of Things (IoT) becoming part of our daily life and our environment, we expect rapid growth in the number of connected devices. IoT is expected to connect billions of devices and humans to bring promising advantages for us. With this growth, fog computing, along with its related edge computing paradigms, such as multi-access edge computing (MEC) and cloudlet, are seen as promising solutions for handling the large volume of security-critical and time-sensitive data that is being produced by the IoT. In this paper, we first provide a tutorial on fog computing and its related computing paradigms, including their similarities and differences. Next, we provide a taxonomy of research topics in fog computing, and through a comprehensive survey, we summarize and categorize the efforts on fog computing and its related computing paradigms. Finally, we provide challenges and future directions for research in fog computing.

The Internet of Things (IoT) is being hailed as the next wave revolutionizing our society, and smart homes, enterprises, and cities are increasingly being equipped with a plethora of IoT devices. Yet, operators of such smart environments may not even be fully aware of their IoT assets, let alone whether each IoT device is functioning properly safe from cyber-attacks. In this paper, we address this challenge by developing a robust framework for IoT device classification using traffic characteristics obtained at the network level. Our contributions are fourfold. First, we instrument a smart environment with 28 different IoT devices spanning cameras, lights, plugs, motion sensors, appliances, and health-monitors. We collect and synthesize traffic traces from this infrastructure for a period of six months, a subset of which we release as open data for the community to use. Second, we present insights into the underlying network traffic characteristics using statistical attributes such as activity cycles, port numbers, signalling patterns, and cipher suites. Third, we develop a multi-stage machine learning based classification algorithm and demonstrate its ability to identify specific IoT devices with over 99 percent accuracy based on their network activity. Finally, we discuss the trade-offs between cost, speed, and performance involved in deploying the classification framework in real-time. Our study paves the way for operators of smart environments to monitor their IoT assets for presence, functionality, and cyber-security without requiring any specialized devices or protocols.

Web-based social networks have become popular as a medium for disseminating information and connecting like-minded people. The public accessibility of such networks with the ability to share opinions, thoughts, information, and experience offers great promise to enterprises and governments. In addition to individuals using such networks to connect to their friends and families, governments and enterprises have started exploiting these platforms for delivering their services to citizens and customers. However, the success of such attempts relies on the level of trust that members have with each other as well as with the service provider. Therefore, trust becomes an essential and important element of a successful social network. In this article, we present the first comprehensive review of social and computer science literature on trust in social networks. We first review the existing definitions of trust and define social trust in the context of social networks. We then discuss recent works addressing three aspects of social trust: trust information collection , trust evaluation , and trust dissemination . Finally, we compare and contrast the literature and identify areas for further research in social trust.

Suboptimal patient selection and recruiting techniques, paired with the inability to monitor and coach patients effectively during clinical trials, are two of the main causes for high trial failure rates.High failure rates of clinical trials contribute substantially to the inefficiency of the drug development cycle, in other words the trend that fewer new drugs reach the market despite increasing pharma R&D investment. This trend has been observed for decades and is ongoing.AI techniques have advanced to a level of maturity that allows them to be employed under real-life conditions to assist human decision-makers.AI has the potential to transform key steps of clinical trial design from study preparation to execution towards improving trial success rates, thus lowering the pharma R&D burden. Clinical trials consume the latter half of the 10 to 15 year, 1.5–2.0 billion USD, development cycle for bringing a single new drug to market. Hence, a failed trial sinks not only the investment into the trial itself but also the preclinical development costs, rendering the loss per failed clinical trial at 800 million to 1.4 billion USD. Suboptimal patient cohort selection and recruiting techniques, paired with the inability to monitor patients effectively during trials, are two of the main causes for high trial failure rates: only one of 10 compounds entering a clinical trial reaches the market. We explain how recent advances in artificial intelligence (AI) can be used to reshape key steps of clinical trial design towards increasing trial success rates. Clinical trials consume the latter half of the 10 to 15 year, 1.5–2.0 billion USD, development cycle for bringing a single new drug to market. Hence, a failed trial sinks not only the investment into the trial itself but also the preclinical development costs, rendering the loss per failed clinical trial at 800 million to 1.4 billion USD. Suboptimal patient cohort selection and recruiting techniques, paired with the inability to monitor patients effectively during trials, are two of the main causes for high trial failure rates: only one of 10 compounds entering a clinical trial reaches the market. We explain how recent advances in artificial intelligence (AI) can be used to reshape key steps of clinical trial design towards increasing trial success rates. It takes on average 10–15 years and USD 1.5–2.0 billion to bring a new drug to market. Approximately half of this time and investment is consumed during the clinical trial phases of the drug development cycle. The remaining 50% of R&D expenditure covers preclinical compound discovery and testing, as well as regulatory processes (Figure 1). Although pharma and biotechnology companies have continuously increased R&D investment for decades, the number of new drugs gaining regulatory approval per billion USD spent has halved approximately every 9 years [1.Scannell J.W. et al.Diagnosing the decline in pharmaceutical R&D efficiency.Nat. Rev. Drug Discov. 2012; 11: 191-200Crossref PubMed Scopus (1252) Google Scholar]. Reversing Moore’s law (see Glossary) from the world of semiconductor technology, this trend has been termed Eroom’s Law. It is ongoingi [2.Thomas D.W. et al.Clinical Development Success Rates 2006–2015. BIO, Biomedtracker, and Amplion, 2016Google Scholar] and poses a severe threat to the existing clinical development business model: in the post-blockbuster drugs era a lack of go-to-market efficiency of that magnitude is not sustainable. One of the main stumbling blocks in the drug development pipeline is the high failure rate of clinical trials. Less than one third of all Phase II compounds advance to Phase III [3.Hay M. et al.Clinical development success rates for investigational drugs.Nat. Biotechnol. 2014; 32: 40-51Crossref PubMed Scopus (1489) Google Scholar]. More than one third of all Phase III compounds fail to advance to approval [4.Wong C.H. et al.Estimation of clinical trial success rates and related parameters.Biostatistics. 2019; 20: 273-286Crossref PubMed Scopus (580) Google Scholar]. Because these crucial checkpoints do not occur until far into the second half of the R&D cycle – with the most complex Phase III trials carrying ~60% of the overall trial costs (Figure 1) – the resulting loss per failed clinical trial lies in the order of 0.8–1.4 billion USDii, thus constituting a significant write-off of the total R&D investment. Two of the key factors causing a clinical trial to be unsuccessful are patient cohort selection and recruiting mechanisms which fail to bring the best suited patients to a trial in time, as well as a lack of technical infrastructure to cope with the complexity of running a trial – especially in its later phases – in the absence of reliable and efficient adherence control, patient monitoring, and clinical endpoint detection systems. AI (Box 1) can help to overcome these shortcomings of current clinical trial design. Machine learning (ML), and deep learning (DL) in particular (Box 2), are able to automatically find patterns of meaning in large datasets such as text, speech, or images. Natural language processing (NLP) can understand and correlate content in written or spoken language, and human–machine interfaces (HMIs) (Box 2) allow natural exchange of information between computers and humans. These capabilities can be used for correlating large and diverse datasets such as electronic health records (EHRs), medical literature, and trial databases for improved patient–trial matching and recruitment before a trial starts, as well as for monitoring patients automatically and continuously during the trial, thereby allowing improved adherence control and yielding more reliable and efficient endpoint assessment. In the following sections we highlight aspects of clinical trial design with immediate potential entry points for AI, and explain specific AI techniques of interest and how their application will improve trial performance (Figure 2, Key Figure).Box 1The Evolution of AIThe use of AI in medicine dates back to the early 1970s when expert systems such as MYCIN were first introduced to provide diagnostic decision support [48.Clancey W.J. Shortliffe E.H. Readings in Medical Artificial Intelligence: The First Decade. Addison-Wesley Longman, 1984Google Scholar]. However, early medical AI systems relied heavily on medical domain experts to train computers by encoding clinical knowledge as logic rules for specific clinical scenarios. Such systems suffered from the limitation that they were labor-intensive and time-consuming to construct, and once built they were rigid and difficult to update [49.McCauley N. Ala M. The use of expert systems in the healthcare industry.Inf. Manag. 1992; 22: 227-235Crossref Scopus (23) Google Scholar]. More advanced ML systems that are capable of training themselves to learn these rules by identifying and weighing relevant features from data such as unstructured text, medical images, and EHRs emerged in the 90s and 2000s, but were relatively slow to be adopted by the medical field, largely because of the lack of widely available data and the fact that the early methods required intense feature-engineering efforts involving serious commitments from medical domain experts [50.Niu F. et al.HOGWILD!: a lock-free approach to parallelizing stochastic gradient descent.arXiv. 2011; (Published online June 28, 2011. https://arxiv.org/abs/1106.5730)Google Scholar].This situation has changed dramatically recently because of two factors. First, the of AI itself in and related ML by and large training datasets et PubMed Scopus Google F. et learning in medicine – and Scopus Google Scholar]. medical data available in to new advances as well as efforts such as the in the years have a in efforts as well as early of AI in from medical for et and of a deep learning for detection of in PubMed Scopus Google Scholar] and et of with deep PubMed Scopus Google to the use of data to clinical from to the of human and artificial 2019; PubMed Scopus Google Scholar]. The of has also from this in AI methods at from natural language processing (NLP) of the et learning for information learning from for Google to et and knowledge from of with 2014; 20: to of of learning with and on Google and drug PubMed Scopus Google in of human intelligence processes and The of AI is to that can the world and in the as ML for between in large databases to help a to the and capabilities of the human from new a between or and to as a a or a learning a of ML methods on artificial by information processing and in that use to level features from The in to the number of which the data is learning learning is of ML that is with that can in as to of and to efficient to this a between a human and a artificial capable of automatically and to spoken or written human language a human–machine learning the study of that a of data to or to the ML is to be a of language processing a of AI with the between computers and human in particular how to computers to and large of natural language from and a of in AI, and at the electronic of of or into text, from a a of a a or from on The use of AI in medicine dates back to the early 1970s when expert systems such as MYCIN were first introduced to provide diagnostic decision support [48.Clancey W.J. Shortliffe E.H. Readings in Medical Artificial Intelligence: The First Decade. Addison-Wesley Longman, 1984Google Scholar]. However, early medical AI systems relied heavily on medical domain experts to train computers by encoding clinical knowledge as logic rules for specific clinical scenarios. Such systems suffered from the limitation that they were labor-intensive and time-consuming to construct, and once built they were rigid and difficult to update [49.McCauley N. Ala M. The use of expert systems in the healthcare industry.Inf. Manag. 1992; 22: 227-235Crossref Scopus (23) Google Scholar]. More advanced ML systems that are capable of training themselves to learn these rules by identifying and weighing relevant features from data such as unstructured text, medical images, and EHRs emerged in the 90s and 2000s, but were relatively slow to be adopted by the medical field, largely because of the lack of widely available data and the fact that the early methods required intense feature-engineering efforts involving serious commitments from medical domain experts [50.Niu F. et al.HOGWILD!: a lock-free approach to parallelizing stochastic gradient descent.arXiv. 2011; (Published online June 28, 2011. https://arxiv.org/abs/1106.5730)Google Scholar]. This situation has changed dramatically recently because of two factors. First, the of AI itself in and related ML by and large training datasets et PubMed Scopus Google F. et learning in medicine – and Scopus Google Scholar]. medical data available in to new advances as well as efforts such as the in the years have a in efforts as well as early of AI in from medical for et and of a deep learning for detection of in PubMed Scopus Google Scholar] and et of with deep PubMed Scopus Google to the use of data to clinical from to the of human and artificial 2019; PubMed Scopus Google Scholar]. The of has also from this in AI methods at from natural language processing (NLP) of the et learning for information learning from for Google to et and knowledge from of with 2014; 20: to of of learning with and on Google and drug PubMed Scopus Google Scholar]. Artificial of human intelligence processes and The of AI is to that can the world and in the as ML for between in large databases to help a to the and capabilities of the human from new a between or and to as a a or a learning a of ML methods on artificial by information processing and in that use to level features from The in to the number of which the data is learning learning is of ML that is with that can in as to of and to efficient to this a between a human and a artificial capable of automatically and to spoken or written human language a human–machine Machine learning the study of that a of data to or to the ML is to be a of Natural language processing a of AI with the between computers and human in particular how to computers to and large of natural language from and a of in AI, and at the electronic of of or into text, from a a of a a or from on clinical trial poses on patients with to and to The medical of a specific patient them patient not be at the of the or to a specific that is by the drug to be thus that patient and patients not be to they they not be of a matching trial or find the recruitment complex and to patients these under recruitment a and is in fact the number one for trial of all trials do not and to one third of all Phase III trials fail to recruitment takes one third of the overall trial Phase III trials of the total costs for a drug all trial phases because they the patient failure rate because of patient recruitment in Phase III trials one of the most severe shortcomings of clinical trial trials with the patient most from patient recruitment and systems can help to improve patient cohort and provide with patient recruitment (Figure Clinical trials are not to the of a in a of the but to a of the in which the of the is can more be a to as trial a patient is a not of the their in the trial will automatically the observed of the drug not be with the of success or absence during the a that not or for patients to to the a high number of patients not success of a trial, but patients the of its In world the of use diagnostic and for and Medical Scholar] to which the drug are in the patient or Although trials which from such approach a relatively of all trials, they also to be the most trials – especially when medical techniques are Hence, in not be a and to be for the of under clinical be methods are to data with electronic medical and other patient and – from to medical – to that to that can be more and thereby and patient This a for and such as (Box 2) to the and of this data from and as a single for the of its is especially in the of data to their and The data of AI them a for data which is key to for clinical trial and However, be to of ML as a of in the training compound testing, and compounds for clinical trials can be by and AI, and techniques N. et intelligence in use of to in PubMed Scopus Google et on of the on and Scopus Google et how can be to data in PubMed Scopus Google Scholar]. a and more efficient for between and than discovery techniques has been et how can be to data in PubMed Scopus Google Scholar]. This allow to be that have a of success during clinical trials, and the of with a of before they the clinical AI and methods can also be used to patient cohort selection one or more of the following by the and Drug by by patients are more to have a clinical also and by identifying a more capable of to a also termed (Figure is a health that on the of identifying patients with specific of The can be as as patients with or as complex as patients with II and of The of electronic is far more than a and methods to for patient data and to complex of clinical domain Although early methods on rules were for they to be for more complex and more In recent years have been increasing efforts to design a diverse of ML from to to (Box 2), that have towards able to complex et in electronic from to learning Rev. PubMed Google Scholar]. Although electronic can be to patient is not to or ML methods are for for key which are or to are by of and which provide information et of 2019; PubMed Scopus Google et of to study of 2019; PubMed Scopus Google Scholar]. more complex that are to and The recently a that advanced a clinical trial for the regulatory at the and the The for and of and early that can be used for clinical trial design et is clinical trial design for PubMed Scopus Google Scholar]. on this ML methods for are to provide and and of complexity and of such as drugs are not available et of in Google et data for in Google et approach and its application to 2019; Scopus Google et with matching for of of the on Google et and contribute to a single in early PubMed Scopus Google Scholar]. The complexity of trial in of number and medical for a patient to and their information from this large and unstructured is a significant that a processing on and patients is this that largely a patient is and to in a and also the recruiting and the patient of AI techniques can with automatically the in the et in deep learning natural language Scopus Google Scholar] can be used to written and spoken language from a of and unstructured data of techniques to clinical trial design is in a recent by with clinical trials that fail and for improving the of a 11: PubMed Scopus Google Scholar]. The and of in the development of learning techniques and the Manag. Scopus Google Scholar] techniques allow content to be into for the human ML et PubMed Scopus Google Scholar] and in particular deep learning (Box 2) systems to learn and on the of their into systems these AI techniques or can be used to automatically and clinical trial find between specific patients and recruiting trials, and these to and patients (Figure Such clinical trial matching systems have been and have their in real-life use et al.Clinical performance for clinical trial matching at Google Because of the AI of these and improved performance will on the and of data which are for development and AI and ML techniques such as and have also been to available content such for trial trial and to automatically potential between trials of and specific patients in their such a patients of trials of interest and allow them to with for of and the first a have been We the of AI will improve the and thus the of such substantially in the The and of data that are used by AI methods are not are for on the one a lack of regulatory on data causes to to be with other or not at and to in a data exchange or the other a to patient data and difficult for patients themselves to their This is as to healthcare systems more and are by and medical towards this M. the of electronic data and Scholar]. In such for the and and the to as and health data complex in the of data and The new health to do with 2019; PubMed Scopus Google The The of in Scholar]. as with for clinical trial matching the aspects of data and as well as a of of AI to be to that systems are and regulatory the patients into a clinical trial is a investment of time and The on this investment can only be of the Hence, is that patients in the trial, to trial and rules the trial, and that all for monitoring the of the drug are and of clinical trials do not patient and the average rate clinical trials is by a lack of adherence to trial recruiting which to trial and of the rate in a trial to in patients required to the of the a study in which half of the patients are of patients to be to the of the patient monitoring and methods during trials can be used to the adherence endpoint detection more and thus and AI techniques in with new to such and patient monitoring systems. We in the sections with adherence patients are required to records of their and of a of other related to their to and This can be and to on average of patients into a clinical trial et to from on the between and drug Rev. 2012; PubMed Scopus Google Scholar]. and monitoring can be used to automatically and continuously patient thereby the patient of this ML and can be used to such data in for and of (Figure This approach allows to be which – because the are with data – to be and to in and patient Such as for adherence or lack and – as or patient is required – will also for endpoint detection more and than current AI also has to in endpoint detection – a that is at ML have been et approach for of in Scholar] – and recently et trial of diagnostic for detection of in PubMed Scopus Google et artificial intelligence for detection in with 2019; (Published online PubMed Scopus Google Scholar] – for for the detection of from medical images. this with that conditions et and of in deep and Medical et medical Scopus Google et deep learning for and in PubMed Scopus Google Scholar] will the with by AI and ML methods also be used to the of for a specific in other words to the of patient that the patient be with to the study (Figure One such the use of deep learning by and learning with for and clinical trial Scholar] to the that with by a ML the at in and the with the and of that to a to that of In trials of the ML that the to a or half of all the the and thus in patient adherence and in and early for allows with patients and the causes of to be for severe or of study and be and before they to The of and is and will to be of the clinical study design. for in and as well as for data from first for and patient monitoring systems have recently been or et artificial intelligence to the of in patients on PubMed Scopus Google Scholar]. The of available with as well as for running advanced on such will allow more to be for a of In a et the and of clinical from advanced diagnostic and other medical health to be the techniques of can in and and in support of clinical trial and to new et of and with health study at the in PubMed Scopus Google Scholar]. We to more the of such on trial efficiency clinical trials in the to in the following we a into a drug trials are the trial and trials [2.Thomas D.W. et al.Clinical Development Success Rates 2006–2015. BIO, Biomedtracker, and Amplion, 2016Google Scholar]. It is to monitor patients in trials for adherence control and endpoint the of of for patients to to control their or to patient absence will not be able to the patient to not and also to not such a from the trial monitoring by medical or not allow reliable in most The for this is that to be in other words they in and for a specific patient their to This the and of a for the a diagnostic for one patient not to and for the patient diagnostic patterns This following a diagnostic and that can be to continuously and patient data for patient as they time, and automatically to in and will be to allow learning from these AI and that AI and can be to such patient monitoring and are to to these and to thus bring medicine to advances in and allow to be to or at the of This from information and into information and – a which we systems to the human and the data on the or to the for the a increasing number of can be and of data for in other words continuously and data in at the of will be to information before the for data or in with data and systems allow this to be The of such (Figure of the following into capable of running with interfaces to which and information on specific and with or for patient and The can thus a a or a of and in of technical maturity have been or are under development et in PubMed Scopus Google Scholar] for into of et patient and PubMed Scopus Google and monitoring, deep with et on a 2019; Scopus Google and data et systems capable of on the as a PubMed Scopus Google et of PubMed Scopus Google and et of for medicine PubMed Scopus Google et for 2019; PubMed Scopus Google in a new The of data which most of these is Although has on data deep recent has that are also to complex et a deep for (Published online et a deep for and 2019; (Published online et learning for the 2019; (Published online Scholar]. continuously in at the of are in AI and AI techniques the years have to of such processing available for real-life These can be into AI such as et million with a and 2014; Scopus Google with AI such as and and which can be AI such as the carrying the and also a of other on of these techniques, the first have emerged in the of for monitoring and and in and and have the of AI for detection et detection systems and their for of PubMed Scopus Google and et data and deep a PubMed Scopus Google is – PubMed Scopus Google The can be patient and automatically to the patterns of patient as they and The monitoring systems of time the for and will thus the and for In other and ML have been used to automatically and to monitor patients in and to of other in and of data and are key for of AI into clinical trial design. The is for AI and in with and pharma have to and best for into clinical trial at the of AI, of and healthcare will more with advanced capabilities for monitoring of patients and M. et data new in and on and is these for improving the performance of trials. It is to that data and a in the and of and regulatory will on the and of patient data and of from the data advanced of monitoring and will for and and exchange of monitoring The of the development for new drugs has pharma and other of clinical development in a et new drugs or existing Rev. Drug Discov. PubMed Scopus (23) Google the era of drugs is to but the R&D for new drugs to the is slow and to for this of the business and of the is for a to a new of and

The Linux Kernel currently supports at least 8 distinct virtualization systems: Xen, KVM, VMware's VMI, IBM's System p, IBM's System z, User Mode Linux, lguest and IBM's legacy iSeries. It seems likely that more such systems will appear, and until recently each of these had its own block, network, console and other drivers with varying features and optimizations. The attempt to address this is virtio: a series of efficient, well-maintained Linux drivers which can be adapted for various different hypervisor implementations using a shim layer. This includes a simple extensible feature mechanism for each driver. We also provide an obvious ring buffer transport implementation called vring, which is currently used by KVM and lguest. This has the subtle effect of providing a path of least resistance for any new hypervisors: supporting this efficient transport mechanism will immediately reduce the amount of work which needs to be done. Finally, we provide an implementation which presents the vring transport and device configuration as a PCI device: this means guest operating systems merely need a new PCI driver, and hypervisors need only add vring support to the virtual devices they implement (currently only KVM does this). This paper will describe the virtio API layer as implemented in Linux, then the vring implementation, and finally its embodiment in a PCI device for simple adoption on otherwise fully-virtualized guests. We'll wrap up with some of the preliminary work to integrate this I/O mechanism deeper into the Linux host kernel.

Detecting COVID-19 early may help in devising an appropriate treatment plan and disease containment decisions. In this study, we demonstrate how transfer learning from deep learning models can be used to perform COVID-19 detection using images from three most commonly used medical imaging modes X-Ray, Ultrasound, and CT scan. The aim is to provide over-stressed medical professionals a second pair of eyes through intelligent deep learning image classification models. We identify a suitable <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Convolutional Neural Network</i> (CNN) model through initial comparative study of several popular CNN models. We then optimize the selected VGG19 model for the image modalities to show how the models can be used for the highly scarce and challenging COVID-19 datasets. We highlight the challenges (including dataset size and quality) in utilizing current publicly available COVID-19 datasets for developing useful deep learning models and how it adversely impacts the trainability of complex models. We also propose an image pre-processing stage to create a trustworthy image dataset for developing and testing the deep learning models. The new approach is aimed to reduce unwanted noise from the images so that deep learning models can focus on detecting diseases with specific features from them. Our results indicate that Ultrasound images provide superior detection accuracy compared to X-Ray and CT scans. The experimental results highlight that with limited data, most of the deeper networks struggle to train well and provides less consistency over the three imaging modes we are using. The selected VGG19 model, which is then extensively tuned with appropriate parameters, performs in considerable levels of COVID-19 detection against pneumonia or normal for all three lung image modes with the precision of up to 86% for X-Ray, 100% for Ultrasound and 84% for CT scans.

Recognizing the layout of unstructured digital documents is an important step when parsing the documents into structured machine-readable format for downstream applications. Deep neural networks that are developed for computer vision have been proven to be an effective method to analyze layout of document images. However, document layout datasets that are currently publicly available are several magnitudes smaller than established computing vision datasets. Models have to be trained by transfer learning from a base model that is pre-trained on a traditional computer vision dataset. In this paper, we develop the PubLayNet dataset for document layout analysis by automatically matching the XML representations and the content of over 1 million PDF articles that are publicly available on PubMed Central. The size of the dataset is comparable to established computer vision datasets, containing over 360 thousand document images, where typical document layout elements are annotated. The experiments demonstrate that deep neural networks trained on PubLayNet accurately recognize the layout of scientific articles. The pre-trained models are also a more effective base mode for transfer learning on a different document domain. We release the dataset (https://github.com/ibm-aur-nlp/PubLayNet) to support development and evaluation of more advanced models for document layout analysis.

We address the problem of cross-domain image retrieval, considering the following practical application: given a user photo depicting a clothing image, our goal is to retrieve the same or attribute-similar clothing items from online shopping stores. This is a challenging problem due to the large discrepancy between online shopping images, usually taken in ideal lighting/pose/background conditions, and user photos captured in uncontrolled conditions. To address this problem, we propose a Dual Attribute-aware Ranking Network (DARN) for retrieval feature learning. More specifically, DARN consists of two sub-networks, one for each domain, whose retrieval feature representations are driven by semantic attribute learning. We show that this attribute-guided learning is a key factor for retrieval accuracy improvement. In addition, to further align with the nature of the retrieval problem, we impose a triplet visual similarity constraint for learning to rank across the two subnetworks. Another contribution of our work is a large-scale dataset which makes the network learning feasible. We exploit customer review websites to crawl a large set of online shopping images and corresponding offline user photos with fine-grained clothing attributes, i.e., around 450,000 online shopping images and about 90,000 exact offline counterpart images of those online ones. All these images are collected from real-world consumer websites reflecting the diversity of the data modality, which makes this dataset unique and rare in the academic community. We extensively evaluate the retrieval performance of networks in different configurations. The top-20 retrieval accuracy is doubled when using the proposed DARN other than the current popular solution using pre-trained CNN features only (0.570 vs. 0.268).

Importance: Mammography screening currently relies on subjective human interpretation. Artificial intelligence (AI) advances could be used to increase mammography screening accuracy by reducing missed cancers and false positives. Objective: To evaluate whether AI can overcome human mammography interpretation limitations with a rigorous, unbiased evaluation of machine learning algorithms. Design, Setting, and Participants: In this diagnostic accuracy study conducted between September 2016 and November 2017, an international, crowdsourced challenge was hosted to foster AI algorithm development focused on interpreting screening mammography. More than 1100 participants comprising 126 teams from 44 countries participated. Analysis began November 18, 2016. Main Outcomes and Measurements: Algorithms used images alone (challenge 1) or combined images, previous examinations (if available), and clinical and demographic risk factor data (challenge 2) and output a score that translated to cancer yes/no within 12 months. Algorithm accuracy for breast cancer detection was evaluated using area under the curve and algorithm specificity compared with radiologists' specificity with radiologists' sensitivity set at 85.9% (United States) and 83.9% (Sweden). An ensemble method aggregating top-performing AI algorithms and radiologists' recall assessment was developed and evaluated. Results: Overall, 144 231 screening mammograms from 85 580 US women (952 cancer positive ≤12 months from screening) were used for algorithm training and validation. A second independent validation cohort included 166 578 examinations from 68 008 Swedish women (780 cancer positive). The top-performing algorithm achieved an area under the curve of 0.858 (United States) and 0.903 (Sweden) and 66.2% (United States) and 81.2% (Sweden) specificity at the radiologists' sensitivity, lower than community-practice radiologists' specificity of 90.5% (United States) and 98.5% (Sweden). Combining top-performing algorithms and US radiologist assessments resulted in a higher area under the curve of 0.942 and achieved a significantly improved specificity (92.0%) at the same sensitivity. Conclusions and Relevance: While no single AI algorithm outperformed radiologists, an ensemble of AI algorithms combined with radiologist assessment in a single-reader screening environment improved overall accuracy. This study underscores the potential of using machine learning methods for enhancing mammography screening interpretation.

Tiny computers located in end-user premises are becoming popular as local servers for Internet of Things (IoT) and Fog computing services. These highly distributed servers that can host and distribute content and applications in a peer-to-peer (P2P) fashion are known as nano data centers (nDCs). Despite the growing popularity of nano servers, their energy consumption is not well-investigated. To study energy consumption of nDCs, we propose and use flow-based and time-based energy consumption models for shared and unshared network equipment, respectively. To apply and validate these models, a set of measurements and experiments are performed to compare energy consumption of a service provided by nDCs and centralized data centers (DCs). A number of findings emerge from our study, including the factors in the system design that allow nDCs to consume less energy than its centralized counterpart. These include the type of access network attached to nano servers and nano server's time utilization (the ratio of the idle time to active time). Additionally, the type of applications running on nDCs and factors such as number of downloads, number of updates, and amount of preloaded copies of data influence the energy cost. Our results reveal that number of hops between a user and content has little impact on the total energy consumption compared to the above-mentioned factors. We show that nano servers in Fog computing can complement centralized DCs to serve certain applications, mostly IoT applications for which the source of data is in end-user premises, and lead to energy saving if the applications (or a part of them) are off-loadable from centralized DCs and run on nDCs.

This work presents an approach for melanoma recognition in dermoscopy images that combines deep learning, sparse coding, and support vector machine (SVM) learning algorithms. One of the beneficial aspects of the proposed approach is that unsupervised learning within the domain, and feature transfer from the domain of natural photographs, eliminates the need of annotated data in the target task to learn good features. The applied feature transfer also allows the system to draw analogies between observations in dermoscopic images and observations in the natural world, mimicking the process clinical experts themselves employ to describe patterns in skin lesions. To evaluate the methodology, performance is measured on a dataset obtained from the International Skin Imaging Collaboration, containing 2624 clinical cases of melanoma (334), atypical nevi (144), and benign lesions (2146). The approach is compared to the prior state-of-art method on this dataset. Two-fold cross-validation is performed 20 times for evaluation (40 total experiments), and two discrimination tasks are examined: 1) melanoma vs. all non-melanoma lesions, and 2) melanoma vs. atypical lesions only. The presented approach achieves an accuracy of 93.1% (94.9% sensitivity, and 92.8% specificity) for the first task, and 73.9% accuracy (73.8% sensitivity, and 74.3% specificity) for the second task. In comparison, prior state-of-art ensemble modeling approaches alone yield 91.2% accuracy (93.0% sensitivity, and 91.0% specificity) first the first task, and 71.5% accuracy (72.7% sensitivity, and 68.9% specificity) for the second. Differences in performance were statistically significant (p $$<$$ 0.05), suggesting the proposed approach is an effective improvement over prior state-of-art.

Campuses and cities of the near future will be equipped with vast numbers of IoT devices. Operators of such environments may not even be fully aware of their IoT assets, let alone whether each IoT device is functioning properly safe from cyber-attacks. This paper proposes the use of network traffic analytics to characterize IoT devices, including their typical behaviour mode. We first collect and synthesize traffic traces from a smart-campus environment instrumented with a diversity of IoT devices including cameras, lights, appliances, and health-monitors; our traces, collected over a period of 3 weeks, are released as open data to the public. We then analyze the traffic traces to characterize statistical attributes such as data rates and burstiness, activity cycles, and signalling patterns, for over 20 IoT devices deployed in our environment. Finally, using these attributes, we develop a classification method that can not only distinguish IoT from non-IoT traffic, but also identify specific IoT devices with over 95% accuracy. Our study empowers operators of smart cities and campuses to discover and monitor their IoT assets based on their network behaviour.

BACKGROUND: Seizure prediction can increase independence and allow preventative treatment for patients with epilepsy. We present a proof-of-concept for a seizure prediction system that is accurate, fully automated, patient-specific, and tunable to an individual's needs. METHODS: Intracranial electroencephalography (iEEG) data of ten patients obtained from a seizure advisory system were analyzed as part of a pseudoprospective seizure prediction study. First, a deep learning classifier was trained to distinguish between preictal and interictal signals. Second, classifier performance was tested on held-out iEEG data from all patients and benchmarked against the performance of a random predictor. Third, the prediction system was tuned so sensitivity or time in warning could be prioritized by the patient. Finally, a demonstration of the feasibility of deployment of the prediction system onto an ultra-low power neuromorphic chip for autonomous operation on a wearable device is provided. RESULTS: The prediction system achieved mean sensitivity of 69% and mean time in warning of 27%, significantly surpassing an equivalent random predictor for all patients by 42%. CONCLUSION: This study demonstrates that deep learning in combination with neuromorphic hardware can provide the basis for a wearable, real-time, always-on, patient-specific seizure warning system with low power consumption and reliable long-term performance.

Systems biology has experienced dramatic growth in the number, size, and complexity of computational models. To reproduce simulation results and reuse models, researchers must exchange unambiguous model descriptions. We review the latest edition of the Systems Biology Markup Language (SBML), a format designed for this purpose. A community of modelers and software authors developed SBML Level 3 over the past decade. Its modular form consists of a core suited to representing reaction-based models and packages that extend the core with features suited to other model types including constraint-based models, reaction-diffusion models, logical network models, and rule-based models. The format leverages two decades of SBML and a rich software ecosystem that transformed how systems biologists build and interact with models. More recently, the rise of multiscale models of whole cells and organs, and new data sources such as single-cell measurements and live imaging, has precipitated new ways of integrating data with models. We provide our perspectives on the challenges presented by these developments and how SBML Level 3 provides the foundation needed to support this evolution.

Ensemble analysis has recently been studied in the context of the outlier detection problem. In this paper, we investigate the theoretical underpinnings of outlier ensemble analysis. In spite of the significant differences between the classification and the outlier analysis problems, we show that the theoretical underpinnings between the two problems are actually quite similar in terms of the bias-variance trade-off. We explain the existing algorithms within this traditional framework, and clarify misconceptions about the reasoning underpinning these methods. We propose more effective variants of subsampling and feature bagging. We also discuss the impact of the combination function and discuss the specific trade-offs of the average and maximization functions. We use these insights to propose new combination functions that are robust in many settings.

The question of whether humans represent grammatical knowledge as a binary condition on membership in a set of well-formed sentences, or as a probabilistic property has been the subject of debate among linguists, psychologists, and cognitive scientists for many decades. Acceptability judgments present a serious problem for both classical binary and probabilistic theories of grammaticality. These judgements are gradient in nature, and so cannot be directly accommodated in a binary formal grammar. However, it is also not possible to simply reduce acceptability to probability. The acceptability of a sentence is not the same as the likelihood of its occurrence, which is, in part, determined by factors like sentence length and lexical frequency. In this paper, we present the results of a set of large-scale experiments using crowd-sourced acceptability judgments that demonstrate gradience to be a pervasive feature in acceptability judgments. We then show how one can predict acceptability judgments on the basis of probability by augmenting probabilistic language models with an acceptability measure. This is a function that normalizes probability values to eliminate the confounding factors of length and lexical frequency. We describe a sequence of modeling experiments with unsupervised language models drawn from state-of-the-art machine learning methods in natural language processing. Several of these models achieve very encouraging levels of accuracy in the acceptability prediction task, as measured by the correlation between the acceptability measure scores and mean human acceptability values. We consider the relevance of these results to the debate on the nature of grammatical competence, and we argue that they support the view that linguistic knowledge can be intrinsically probabilistic.

We address the problem of describing people based on fine-grained clothing attributes. This is an important problem for many practical applications, such as identifying target suspects or finding missing people based on detailed clothing descriptions in surveillance videos or consumer photos. We approach this problem by first mining clothing images with fine-grained attribute labels from online shopping stores. A large-scale dataset is built with about one million images and fine-detailed attribute sub-categories, such as various shades of color (e.g., watermelon red, rosy red, purplish red), clothing types (e.g., down jacket, denim jacket), and patterns (e.g., thin horizontal stripes, houndstooth). As these images are taken in ideal pose/lighting/background conditions, it is unreliable to directly use them as training data for attribute prediction in the domain of unconstrained images captured, for example, by mobile phones or surveillance cameras. In order to bridge this gap, we propose a novel double-path deep domain adaptation network to model the data from the two domains jointly. Several alignment cost layers placed inbetween the two columns ensure the consistency of the two domain features and the feasibility to predict unseen attribute categories in one of the domains. Finally, to achieve a working system with automatic human body alignment, we trained an enhanced RCNN-based detector to localize human bodies in images. Our extensive experimental evaluation demonstrates the effectiveness of the proposed approach for describing people based on fine-grained clothing attributes.

This paper presents a novel computer-aided diagnosis system for melanoma. The novelty lies in the optimised selection and integration of features derived from textural, borderbased and geometrical properties of the melanoma lesion. The texture features are derived from using wavelet-decomposition, the border features are derived from constructing a boundaryseries model of the lesion border and analysing it in spatial and frequency domains, and the geometry features are derived from shape indexes. The optimised selection of features is achieved by using the Gain-Ratio method, which is shown to be computationally efficient for melanoma diagnosis application. Classification is done through the use of four classifiers; namely, Support Vector Machine, Random Forest, Logistic Model Tree and Hidden Naive Bayes. The proposed diagnostic system is applied on a set of 289 dermoscopy images (114 malignant, 175 benign) partitioned into train, validation and test image sets. The system achieves and accuracy of 91.26% and AUC value of 0.937, when 23 features are used. Other important findings include (i) the clear advantage gained in complementing texture with border and geometry features, compared to using texture information only, and (ii) higher contribution of texture features than border-based features in the optimised feature set.

Optical coherence tomography (OCT) based measurements of retinal layer thickness, such as the retinal nerve fibre layer (RNFL) and the ganglion cell with inner plexiform layer (GCIPL) are commonly employed for the diagnosis and monitoring of glaucoma. Previously, machine learning techniques have relied on segmentation-based imaging features such as the peripapillary RNFL thickness and the cup-to-disc ratio. Here, we propose a deep learning technique that classifies eyes as healthy or glaucomatous directly from raw, unsegmented OCT volumes of the optic nerve head (ONH) using a 3D Convolutional Neural Network (CNN). We compared the accuracy of this technique with various feature-based machine learning algorithms and demonstrated the superiority of the proposed deep learning based method. Logistic regression was found to be the best performing classical machine learning technique with an AUC of 0.89. In direct comparison, the deep learning approach achieved a substantially higher AUC of 0.94 with the additional advantage of providing insight into which regions of an OCT volume are important for glaucoma detection. Computing Class Activation Maps (CAM), we found that the CNN identified neuroretinal rim and optic disc cupping as well as the lamina cribrosa (LC) and its surrounding areas as the regions significantly associated with the glaucoma classification. These regions anatomically correspond to the well established and commonly used clinical markers for glaucoma diagnosis such as increased cup volume, cup diameter, and neuroretinal rim thinning at the superior and inferior segments.

Energy storage systems have the potential to deliver value in multiple ways, and these must be traded off against one another. An operational strategy that aims to maximize the returned value of such a system can often be significantly improved with the use of forecasting - of demand, generation, and pricing - but consideration of battery degradation is important too. This paper proposes a stochastic dynamic programming approach to optimally operate an energy storage system across a receding horizon. The method operates an energy storage asset to deliver maximal lifetime value, by using available forecasts and by applying a multi-factor battery degradation model that takes into account operational impacts on system degradation. Applying the method to a dataset of a residential Australian customer base demonstrates that an optimally operated system returns a lifetime value which is 160% more, on average, than that of the same system operated using a set-point-based method applied in many settings today.