Can Computing Fabrics really operate over the WAN and
A: The broadband will constitute
an interconnect with which to "loosely couple" cells,
clusters of nodes that are local and tightly coupled
themselves. This will require predictability of latencies
for utmost performance, but that can be achieved through
QoS measures debuting over the next 5 years. Systems
will not use hardware approaches to tightly couple over
broadband connections. It is the essence of fabrics
that they utilize both types of couplings, tight and
loose, but in a more flexible manner than current networks
Won't the additional cost of Computing Fabric nodes
and interconnects, compared with desktop PCs and Ethernet,
slow their adoption?
A: Employees throughout
corporate America may have little to say about what
desktop gets purchased for them – even less if the
functional equivalent, a fabric node, no longer sits
on their desk! TCO was picked up and embraced mighty
fast and if Computing Fabrics bear out then the cost
savings they’ll accrue through more effective use of
ambient cycles will justify purchasing the machines
that save the most money in the final analysis, not
just the cheapest machines.
Besides this, some cells
will be formed using Automated Transparent Object Distribution
(the strategy behind Microsoft's Millennium), requiring
little more than today's desktops.
How will security be handled on Computing Fabrics?
A: This is a big issue
with many sub-issues. For example, what does it mean
to secure a remote memory read or write? How would you
deal with the overhead and still maintain ultra-low
latencies and preserve cache coherence? However, there
are plenty of benefits here to justify the R&D to
attack these problems, as well as sufficient time and
dollars to do so. Tight coupling using software at the
object level will incur fewer security issues than tight
coupling with hardware at the memory page level.
Will Computing Fabrics support legacy compatibility?
A: Existing multithreaded
applications should run on Computing Fabrics, if not
fully exploit their unique properties of dynamic reconfigurability
and fluid system boundaries. Beyond this there are issues
such as incorporating equipment currently owned or purchased
over the next several years (which will be legacy within
Computing Fabrics' evolutionary time frame) into the
fabric. Cellular IRIX from SGI will support both CORBA
and DCOM thus making fabrics employing IRIX interoperable
with most legacy systems.
The enterprise seems a likely place for Computing Fabrics
to take hold, but are fabrics really likely to take
off in the consumer space?
A: The enterprise is where
fabrics will initially be taking off, as an outgrowth
of the current interest in clustering, especially as
the problems that clustering is targeted at have been
so poorly addressed by most vendors.
Moving beyond the enterprise,
into the consumer realm, ones finds what’s being planned
by telcos for residential neighborhoods, distributing
massive processing power through them, heavily utilizing
Now add to that certain
technologies that impact the human interface for consumers,
including 3D, media integration, and the convergence
of devices such as game consoles, set tops, and PCs.
These will be a strong motivator for Computing Fabrics
outside of the enterprise because they significantly
advance the user experience, and they eat up lots of
computational resources, many of which need to be near
the end user because of the speed of light.
Visualize the progression
of Computing Fabric as beginning with many smaller fabrics,
first in the enterprise, that in time join up to become
fewer, larger fabrics. Neighborhoods of processors and
memory within these fabrics exhibit a single system
image through tight coupling. These neighborhoods of
tight coupling are themselves loosely coupled with each
other. And due to the distributed OSes and interconnects
these fabrics will employ, the boundaries of the neighborhoods
will not be rigid but fluid. Feeding and supporting
all this will be economies of scale and reuse far greater
than today’s. These same principles will in time apply
to processors distributed throughout residential areas.
Are Computing Fabrics just about scalability?
A: Computing Fabrics address
scalability but that’s only one small slice of a wide
panorama. Improvements in scalability are largely a
quantitative change. Computing Fabrics will be a qualitative
change (as well as quantitative) in that we’re no longer
talking about networking fixed-architecture systems
but seeing system architecture converge with network
architecture, with architecture itself ultimately becoming
a dynamic variable - Architecture On Demand.
Are Computing Fabrics really "in sight"?
A: When is a new era of
technology within sight? Technologies often first enter
into classified "black" projects long before the public
even gets a whiff of them. Then they enter unclassified
military usage, then on into academic research, on into
the high-end of the commercial marketplace, and only
after many years do they enter the mainstream. At which
point are they "near us", "within sight", or "upon us"?
The intelligence community would say the next era is
upon us while many in education are nowhere near client/server!
The technologies of Computing Fabrics have been and
continue to be proven and are also being scaled. Within
two years they will be implemented with mass-market
Isn't it a well established fact that NUMA machines
have higher latencies than SMP machines do, and always
A: This is not just an
architectural issue, it is also involves engineering
and implementation factors. The normative latency of
some SMPs exceeds the worst case latencies of some NUMA
implementation. The team responsible for Craylink at
SGI/Cray Research has a project to extend Craylink with
three major milestones. Within 2 years time they will
expand the system size that Craylink can support by
extending the reach of Craylink cables to encompass
a very large room. In three years time (from the present
- late 1998) they will expand the bandwidth of Craylink.
And in 5 years time SGI will extend the range again,
this time dramatically, to support distribution throughout
a building and beyond.
One possible implementation
to achieve this is to add a cache coherency protocol
on top of the follow-on to SuperHIPPI, but that is just
one direction being considered. These three extensions
to Craylink are being pursued in parallel by SGI.
Another approach, being
pursued by Microsoft in their Millennium research project,
automates the distribution of COM+ objects around a
network (unlike today where programmers must decide
the location of client and server objects). Furthermore,
they are layering DCOM over VI Architecture links to
achieve very low latencies for remote method invocations.
HIPPI-6400 will support distances up to 10km. With VIA
running over HIPPI-6400 and DCOM over VIA, and Millennium's
Continuum automating the distribution of COM+ objects,
a distributed object space (as contrasted with distributed
shared memory) could reach out over far more than a
quarter of a mile.
Will special programming tools or paradigms be required,
like MPI or PVM?
A: Programming a Computing
Fabric will most likely resemble programming a distributed
shared memory machine. This involves great thread packages
and great compilers, just as it does for programming
centralized shared memory machines (SMPs), although
there are differences in the application on these two
MPI (Message Passing Interface)
and PVM (Parallel Virtual Machine) are not required.
MPI and PVM are used by programmers of massively parallel
machines (and on networks of workstations where supported
by a utility) to obtain portability between parallel
architectures and implementations. They explicitly support
programming distributed "Non-Shared" memory computing,
where each processor has its own memory with it own
address space and message passing is used to coordinate
function invocations, reads, writes, etc. amongst the
ensemble. PVM was developed at Oak Ridge National Lab
in ‘89 to run across a network of UNIX boxes. MPI began
at a workshop on message passing in ’92 and the first
version was published a little over a year later, making
its big debut at Supercomputing ’93 in November of that
Computing Fabrics is not
network computing, which primarily means distributed
file systems and more recently distributed objects.
What we’re moving into is a convergence of the loosely
coupled programming model of networks and MPP (distributed
processing) with the more abstract, transparent programming
of SMPs, where the programmer is shielded from many
of the details of the underlying distribution, since
all memory is shared. It should be pointed out that
the SGI Origin 2000, a precursor of a Computing Fabric,
can also be programmed with a parallel library, such
as MPI, when message passing best suits the needs of
I haven’t heard anyone talk about dynamic reconfiguration
especially in the sense of interconnects since that
could disrupt the entire adaptive routing algorithms
used for these type of systems causing all sorts of
hotspots and contention especially if the applications
didn’t have the same requirements, which they most probably
A: The architecture of
the SGI Origin2000 and its routing system was designed
so that hardware makes routing decisions while software
can reconfigure the network. Right now this reconfiguration
is limited to avoiding faults and better using resources,
but this will be changing. Microsoft is also working
on this as part of the Millennium project at Microsoft
Research, very much focused on creating fluid system
boundaries. These elements differentiate Computing Fabrics
from mere networks of bus- and switch-based multiprocessor
systems with their inherently rigid system boundaries.
Don't Beowulf systems have a great deal of similarity
to Computing Fabrics? I understand these systems have
even enabled screensavers that can detect idle times
automatically and log your machine into the "fabric"
and begin work on a shared problem.
A: Though focused on Linux,
Beowulf is a great project but like Linux is not a commercially
backed one, limiting its applicability in the enterprise.
There is even a 1,000 processor Beowulf system under
construction using Alphas. Beowulf systems do not even
utilize a low-latency link between systems, because
they are intended to exploit technology that is commodity
"today". Also, software tends to follow hardware, often
by significantly long stretches. Today we’re seeing
the beginnings of the hardware for Computing Fabrics.
It will motivate the development of the software. Beowulf
is a software solution to provide some measure of distributed
processing on today’s commodity hardware.
How much will it cost to put a a scalable interconnect
on each desktop in order to create a Computing Fabric
across an enterprise? If it is cheaper to use a non-scalable
one I suspect that one will win. And do we really need
a hypercube on every desk?
A: Despite the fact that
in the mid eighties Danny Hillis suspected that the
Connection Machine would spawn desktop hypercubes in
10 years, no, we do not need a hypercube on the desktop,
that would be a perversion of the direction technology
is headed. Rather, the machines that replace desktops
will participate in a fabric that has hypercube regions
to its topology. At first, say 3-5 years off, it will
indeed be pricey to use cache coherent scalable interconnects.
Now, although NCs haven’t been a big hit (and rightly
so) TCO has caught on, and so too will TCC, Total Cost
of Cycles. This could well make widespread use of modularly
scalable interconnects very attractive as a way of exploiting
an organization’s cycles. Besides this, there are several
human interface directions at work that mandate significant
power (processing cycles and cache) out near the users
but not dedicated to a specific user 24x7x365. These
will weigh heavily as motivation towards distributed
shared processing and Computing Fabrics.
Is all this really necessary to run Word?
A: Ever increasing cycles
will be needed to enhance the human-computer interface,
and that power better be "near" the user though it needn’t
be on their desktop. Desktops themselves may disappear
(literally the desk tops – the computers will depart
with them. More workers will be peripatetic and mobile,
as well as working from home and in the field). The
first place fabrics will catch on is in the heart of
the enterprise, an evolution of server farms. But financial
pressure will likely cause the assimilation of whatever
succeeds the desktop. Two kinds of connectivity will
be present (at least). Information appliances and personal
interface devices will connect in using wireless RF
and scattered IR (and other technologies from DARPA
projects). These will certainly not support cache coherent
SSI but only loosely coupled distributed processing.
Its what’s behind the walls that’s likely to become
tightly coupled but distributed so as to remain relatively
close to the users.
Will Computing Fabrics create a totally homogeneous
computer architecture from the bottom of the industry
to the top?
A: There will still be
layering in the industry and technological innovations
specific to the realm of problems being solved. But
the similarities will be greater than the differences,
that fabric clusters in academia operating at teraflops
will be roughly equivalent to small fabrics in the enterprise,
but the superfabrics will use processor variants with
bigger caches, perhaps extended superscalar architecture,
the latest, fastest SuperDuperHIPPI, while the small
fabrics at the department level will use last year’s
variants. The point is that for the first time these
technologies are variants of one another, not altogether
different beasts. It means that one year’s superfabric
technology can directly become a commodity fabric technology
within years, not decades.
How do Computing Fabrics in residential neighborhoods
help manufacturers get closer to the consumer?
A: When the consumer is
literally embedded in massive processing many new things
become possible. The heart and soul of a company, caught
in its 3D multimedia knowledgebase, can be locally instantiated
for the consumer, allowing them to configure the products
and services based on the production and support capabilities
of the vendors or a cooperative of vendors. Such vendors
will virtually "fuse" their demo, warehousing, and production
spaces with the consumers’ space, sometimes multiple
vendors at a time in a comparison shopping and bid situation.
This is "getting close" to the consumer and demands
lots of power. Do you need this power to send the customer
an invoice? No, but that’s not the kind of cozying up
Why should business begin making plans now for Computing
Fabrics when they’re still years away?
A: Infrastructure, tooling,
training, corporate structure, and capital investment
costs big, very big, and can chain a company so tightly
to the past that they can’t get free of it. Second,
business should consider fabrics in their business model
(analyses and planning – not taking immediate action),
which does look out beyond next Christmas. Technology
vendors will be the first who will need to come to grips
with fabrics so that they can exploit the trend rather
than become a casualty of it.
In retrospect, would organizations
have wanted to become aware of the PC back in the 70’s,
would minicomputer vendors have wanted to know about
distributed processing based on commodity microprocessors,
and would businesses have wanted an advance warning
of the coming web? The answers are trivially easy: yes,
yes, and yes.
Won't Computing Fabrics run into memory locality problems
and contention that their distributed shared memory
architecture can't address?
A: The Computing Fabrics
landscape will likely have very large ensembles of processors,
probably making today’s 64 and 128 processor machines
seem diminutive. So in this future let’s compare a cluster
of 16 Sun Starfire SMP servers, each with 64 processors,
to a single SGI Origin with 1,024 processors. If you’ve
got a problem that fits in the address space of a single
one of the Sun servers then you can claim uniform latencies.
But many of these huge arrays will handle huge problems
that only work nicely in a contiguous address space.
First, these are not going to run on the Sun, as the
problem must be broken up into parts that can be distributed
to each node (of 64 processors) in the Sun cluster.
So let’s say you do just that. What about the latency
of travelling from a processor on one SMP to a processor
on a different SMP in the cluster? Is this latency going
to be uniform with the latency across the crossbar switch
in a single Sun server? No it won't. In fact, its likely
to far exceed the end-to-end latency in the 1,024 processor
Origin. So, if we say this is bad for the Origin then
we’ll have to say it’s doubly bad for the Sun, which
carries over to all loosely coupled clusters of SMPs.
All architectures make
design decisions that invoke tradeoffs. The question
is not if a particular architecture has deficiencies,
they all do, but rather does an architecture make wise
tradeoffs given the kinds of applications it is known
in advance that it will be used for, as well as the
many areas that it will ultimately be applied to. In
pursuit of modularly extendable fabrics SMP just doesn’t
cut it, except at the nodes of the architecture, with
hardware and software used to maintain cache coherence
between these SMPs. This is fine because today’s clusters
of Small SMP’s are likely to evolve with fabric technology
to become exactly what the SGI machines are becoming
– commodity processors and all, but they’ll be lots
cheaper. That’s one of the main points here, as the
Origin architecture goes Intel similar functionality
will come to clusters of commodity machines. This is
big news for the future of computing, as clusters become
systems and systems become clusters, and the distinction
disappears. What’s the source of this revolution? It’s
the 3 key technologies identified herein, the distributed
shared architecture, the rich modularly scalable interconnect,
and the Cellular OS, on their way into Intel space.
code, if you mean code that does not take advantage
of a multithreaded package it won't function on an SMP
either. Effective multithreading is all that’s "minimally"
required to utilize an SMP, a CC-NUMA machine, or a
Computing Fabric. Now if by parallelization you mean
instead rearchitecting software to explicitly use MIMD,
SIMD, or vector parallelization of loops – that’s what
almost everyone is trying to avoid and why a single
address space, whether in an SMP or in CC-NUMA, is so
desired. Programming to a message passing model is vastly
different than programming to a shared memory model,
whether that memory is in reality centralized or distributed.
Programs based on message passing can be run sub-optimally
on an SMP, run less sub-optimally on some NUMA architectures,
and run very well on clusters and MPPs. Programs explicitly
written to an SMP model can run sub-optimally on CC-NUMA
but not at all on a cluster or MPP. Since programs such
as RDBMSes are for the most part (but not entirely)
programmed to the SMP model they will be better supported
and more easily ported to a fabric that supports distributed
shared memory (e.g., ccNUMA) than a message passing
cluster or MPP (note Oracle’s multi-year painful experience
in porting to the nCUBE2 MPP from a code base optimized
for Sequent SMP). Lastly, on contention and hotspotting,
these are minimized by the Cellular OS as well as the
hardware that provides cache coherency. To reiterate,
no architecture solves all problems free of side effects.
SMPs don’t. ccNUMA doesn’t. MPP sure doesn’t. Moving
to a loosely coupled approach using distributed objects
doesn’t either, in fact that only invokes multiple passes
through the entire protocol stack unless layered on
ST or VIA – talk about contention!
How does the SGI Spider chip enable Craylink to function
without contention and without running into an N-squared
increase in system cost and complexity?
A: The Spider chip itself
is non-blocking. Since each of its 6 ports supports
4 virtual channels there can be contention in that a
packet with a recent age may wait while the on-chip
arbiter enables an older packet to progress though the
crossbar. This is a design decision and alternatives
are being explored with the next version of the Spider
chip which will also offer an increased number of ports
(exact increase not yet disclosed). The fact that to
create a totally non-blocking NxN switch requires on
the order of N squared Spider chips is not relevant
to the design of the SGI Origin as it does not use this
architecture - it employs a variant of a hypercube.
The cost of the system
does scale close to linearly with a single discontinuity
when expanding from 64 to 128 processors, then the number
of Spider chips required resumes a linear increase.
In terms of backplane connections between Spider chips
these too increase in a direct linear relationship to
the number of processors. Only the number of Craylink
cables between Spider chips departs from strict linearity,
but only slightly so, in that doubling the number of
processors with this architecture from 128 to 256 ups
the required Craylink count from 112 to 236 rather than
224 – a minor departure. While bandwidth per processor
remains constant as the system is grown it is true that
latency increases between widely separated processors
– this is still NUMA after all and that is a design
tradeoff. However, actual measured latencies for widely
separated processors in a large Origin 2000 are actually
less than the "constant" latency between processors
and memory in many SMPs.
Don’t hypercubes of the SGI Origin design decrease system
bandwidth with increasing size, meaning a fabric based
on this design cannot adequately scale?
A: The hierarchical fat
hypercube topology does not "scale down" as nodes are
added, provided that the metarouters used by the system
between hypercubes increase in dimension as the system
grows – precisely the pattern followed by SGI engineers.
For example, taking a 1,024 processor system built with
this architecture and cutting it in two (bisection)
yields 8 processors per connection, exactly the same
bisection bandwidth as a similarly connected system
of 16 or 32 processors. The key is to increase the dimensionality
of the metarouter as the system grows. For example,
a 128 processor Origin uses 8 2D metarouters (essentially
rings), but a 1,024 processor Origin will use 8 5D Hypercubical
metarouters. It should added that this use of metarouters
will be advantageous when the interconnects are distributed
through a building, though that’s 5 years off.
As for bristling (or the
similar topology of cube connected cycles) this does
indeed impact bandwidth per processor, hence it is wise
to decide on the bristling factor before making design
decisions on the network. It appears that SGI has indeed
done their homework here. The bristle factor (the number
of processors supported by each router) begins at 4
processors per router for configurations of the Origin2000
up to and including a 64 processor configuration. But
beginning with 128 processors (and up once they begin
shipping) the bristling decreases to on average 2 processors
per router. Why? Because the metarouters proportionately
expand, with Spider chips in the metarouter joining
up with Spider chips on node boards to form virtual
super routers of the fat hierarchical hypercube. Bottom
line: bandwidth scales linearly with system size (with
one discontinuity), latency is non-uniform by architectural
choice, and cost scales so close to linearly with size
that it deserves to be called linear scaling.