In the ISO 1990 C standard library, what was the rationale for having memcpy return one of its inputs?

Question

The function memcpy is defined as:

void* memcpy(void* s1, void* s2, size_t n)

and the 1990 ISO standard (ISO 9899:1990) defines the function as:

Description

The memcpy function copies n characters from the object pointed to by s2 into the object pointed to by s1. If copying takes place between objects that overlap, the behaviour is undefined.

Returns

The memcpy function returns the value of s1

But...why? s1 itself doesn't get modified. It seems that newer C standard inherited this. So I'm wondering why the original authors of the standard made this function return anything at all, and why it returns what it does.

This is just a hunch, but it could be because the string manipulation functions already did this (strcpy) etc. That just changes the question to “… the rationale for having strcpy return one of its inputs?” — Stephen Kitt, Jun 21 '22 at 14:16
I suppose so, yes. I'm reading "The Standard C Library" by Plauger (written in 1990 or so), and he does go into some very interesting detail about some of the stdlib stuff and why the committee, of which he was a part, chose to do what they did. It is very amusing to read what he thinks about errno. But things like this are just taken for granted. — Michael Stachowsky, Jun 21 '22 at 14:24
I really enjoyed that book. Intriguingly, the K&R C book (2nd edition) discusses void strcpy(char *s, char *t) (no return value) and then just says “The strcpy in the standard library (<string.h>) returns the target string as its function value.” — Stephen Kitt, Jun 21 '22 at 15:16
For the curious, Plauger's choice words for errno and errno.h can be found here: https://archive.org/details/cprogramming0000koch/page/47/mode/1up — Jim Nelson, Jun 21 '22 at 18:02
A tip on understanding almost anything in the C standard: Many C implementations existed for years before the language was standardized. For the most part, the standard writers simply chose which aspects of existing implementations to codify, with a distinct desire to break as little existing code as possible. Many things were simply inherited from existing implementations and weren't purposefully or intentionally designed by the standards committee. — bta, Jun 22 '22 at 19:32
What Plauger says about errno is hardly controversial, is it? A global variable to say what error somehow happened once upon a time? Bleurgh. Better use a Rust-style Result type. In C you could kinda emulate it with some struct containing either the result of the function call, or an error code. — Omar and Lorraine, Jun 23 '22 at 09:10
@bta: Another tip, which I wish compiler writers would recognize, is that "Undefined Behavior" was catch-all for non-portable constructs whose behavior might be unpredictable on some platforms, but which most implementations were expected to usefully process "in a documented manner characteristic of the environment" or, at worst, by choosing in Unspecified fashion from among a certain range of actions which the environment documented as being characteristic to it (e.g. if a platform's normal ADD instruction traps on overflow, but its INC instruction does not, characterizing integer overflow... — supercat, Jun 23 '22 at 17:11
...as "Implementation Defined" would have strongly implied that a compiler for that platform should either perform signed additions in a manner that would never trap, else refrain from optimizing something like "x+=1;" to use an INC instruction; the authors of the Standard didn't want to forbid that kind of optimizations, but that doesn't mean that they intended that implementations for quiet-wraparound hardware interpret the fact that certain inputs would result in integer overflow as an invitation to process such inputs in gratuitously nonsensical fashion). — supercat, Jun 23 '22 at 17:16
@supercat: Many compiler / library writers want to actively discourage people from exploiting non-standard behavior, as that in turn means that future versions of the compiler / library will have to be consistent in how they define such behavior (reducing the leeway implementors have), or break compatibility with older code (always a very bad thing to do). I understand why this can be infuriating, and I also agree that in some cases this has been taken too far. But generally speaking, I agree with the crede "stay away from UB". — DevSolar, Jun 24 '22 at 13:20
I believe that this question on the StackOverflow is related: https://stackoverflow.com/questions/2723686/c-memcpy-return-value — introspec, Jun 24 '22 at 13:48
@DevSolar: Programming often involves trade-offs between portability and performance. If the authors of the Standard had been seeking to strike a reasonable balance between portability and performance, rather than merely giving programmers a "fighting chance" (their words!) to write portable programs, they would have not left so many things as completely unbounded Undefined Behavior. — supercat, Jun 24 '22 at 14:38
@supercat: There was no balance to strike. The standard committee was not in the business of programming in the first place. The mission was setting a common standard for all the already existing implementations, to avoid the language drifting apart into countless implementation-specific dialects. Another part of the mission was to make it possible to come up with a compliant implementation on any platform. (Well, perhaps with the exception of MVS...) This was not something born out of a laboratory. This was trying to fence in something that had already escaped into the wild long ago. — DevSolar, Jun 24 '22 at 15:37
@DevSolar: When the Standard was recognized as merely setting a baseline dialect which implementations intended to be suitable for low-level programming tasks on various platforms would extend in platform-appropriate fashion, typically by processing constructs "in a documented manner characteristic of the environment", that was useful. It becomes worse than useless, however, when compiler writers treat the phrase "non-portable or erroneous" as synonymous with "non-portable, and therefore erroneous". If one has a 32-bit aligned pointer to a pair of 16-bit integers, and wants... — supercat, Jun 24 '22 at 16:15
...to invert all the bits in both of them, the most efficient way to do that on most current platforms would be to perform a 32-bit read, invert all the bits, and then a 32-bit write. While some compilers may convert void flip_bits(unsigned short *p) { unsigned t; memcpy(&t, p, 4); t ^= 0xFFFFFFFF; memcpy(p, &t, 4); } into code that uses a 32-bit load and store, the Standard was never intended to imply that programmers jump through such hoops rather than writing code that can be straightforwardly translated into the required machine operations. — supercat, Jun 24 '22 at 16:21
@DevSolar: In short, I think there's an attitude that the Committee was trying to describe the maximal subset of features and guarantees programmers should need, even when exclusively targeting halfway-modern platforms, when in reality it was seeking to define the minimal subset of features and guarantees that even the most limited and quirky implementations should be required to provide. — supercat, Jun 24 '22 at 16:24

score 34 · Accepted Answer · edited Jun 23 '22 at 23:18

In early versions of the C language, every function would return something, whether or not the caller would make use of the returned value. Generally, the return value of a function would be whatever happened to be in some particular register of the appropriate type. If code exited a function without making any effort to set the register to something meaningful, and calling code ignored the contents of the register in question, having the function nominally return a meaningless value was simpler and easier than providing a means of having functions not return a value.

I don't think any particular thought was put into the question of what functions like memcpy, strcpy, or strcat should return, but the authors of the Standard didn't want to simply leave the return value unspecified. Since there may have been platforms where functions that don't return a value would be processed differently from those that do, giving such functions a void return type could have broken code that calls the functions without including the appropriate standard header.

I don't think any particular effort was made to have the functions return the most useful value. More likely, the authors of the Standard wanted to have the functions return some specified value, and so they somewhat arbitrarily picked a value to be returned.

I don't see it as arbitrary. In my view, strcpy and memcpy are analogous to assignment operators. If a=56*3, as an expression, returns the value of a, it makes some philosophical sense for strcpy or strcat to return the new string, and for memcpy to return the newly filled buffer. — Nimloth, Jun 21 '22 at 19:49
@Nimloth: It is a consistent but bad design choice for strcat and strcpy, though, discarding the string-length / end-pointer information they had to compute during the operation. This is why stpcpy exists (in POSIX and various other implementations, but not ISO C): My answer on strcpy() return value took some guesses at possible asm / compiler / code-style reasons for the design decision. — Peter Cordes, Jun 23 '22 at 03:09

score 21 · Answer 2 · answered Jun 21 '22 at 20:03

21

So that you can write

s1 = memcpy (s2, memcpy (s3, memcpy (s4, s5)));

which is probably not particularily useful (for string concatenation using strcat, it is, however, and the memxxx and strxxx functions use aligned function signatures).

answered Jun 21 '22 at 20:03

tofro

34,832
4
89
170

3

For string concatenation, using strcat(strcat(strcat(dest, string1), string2), string3) is gratuitously inefficient. I see no reason language or libraries should make gratuitously inefficient constructs more convenient than equally simple but more efficient ones. – supercat Jun 21 '22 at 20:20
@supercat Agreed when you don't need the intermediates updated. But otherwise, you can't get it much cheaper. – tofro Jun 21 '22 at 20:23
11

A function which behaved like strcat, except returning the address of the terminating zero byte that was written to the destination, would support the same chaining syntax efficiently. – supercat Jun 21 '22 at 20:36
3

One possibly-useful construct could be return memcpy(s1, s2) to avoid a register load (before compilers became clever enough). But AFAICT the return value of strcpy and strcat is never used in Unix V7, so I’m not convinced such constructs were a factor. – Stephen Kitt Jun 22 '22 at 04:49
Aren’t memcpy and strcat only looking similar because you omitted the size argument? – Holger Jun 22 '22 at 08:06
5

I often use p = memcpy(malloc(whatever), x, whatever) when I don't care about failling allocations. – Patrick Schlüter Jun 22 '22 at 08:30
1

@supercat "except returning the address of the terminating zero byte that was written to the destination, would support the same chaining syntax efficiently." Joel Spolsky talks about this in his article, Back to Basics: "How do we fix this? A few smart C programmers implemented their own mystrcat as follows: ... At very little extra cost we’re returning a pointer to the end of the new, longer string." – Joshua Taylor Jun 22 '22 at 12:10
@JoshuaTaylor: To the contrary, on most implementations, for less cost we're returning a pointer to the end of the new longer string. – supercat Jun 22 '22 at 13:29
@supercat I agree, depending on the implementation, it's probably even less work (no need to keep around the start pointer). But either way, it wasn't my claim -- I was just quoting the article. :) – Joshua Taylor Jun 22 '22 at 15:49
@JoshuaTaylor: In any event, my point was that I don't think any real thought was put into the question of what such functions should return, though if I were writing a family of strcpy/strcat-like functions, I think the main workhorse would accept as an argument not the size of the destination buffer, but rather a pointer "just past" the end of it, so that multiple concatenation operations could all pass that same pointer address without having to keep track of how much space remained. – supercat Jun 22 '22 at 16:05
@JoshuaTaylor: I'd also probably specify that passing a null-pointer as a destination would return null with no side effects, and that if a string buffer is too small to store the whole string the function would return null. Code that wants to know if a group of concatenated strings all fit could thus simply check whether the function returned a non-null result. – supercat Jun 22 '22 at 16:07
1

@supercat a ponter just past a buffer isn't something I'd consider valid C (and I think the standard agrees with me here) - That's walking on shaky grounds. – tofro Jun 22 '22 at 16:58
4

@tofro: To the contrary, the Standard explicitly specifies that adding one to the address of the last element of an array will yield a pointer that cannot be directly dereferenced, but it it may be used in both relational and equality comparisons, may have a negative value added to it to yield a valid pointer to an array element, and may be used with the pointer-difference operator to measure the distance from other pointers into or just past the array. – supercat Jun 22 '22 at 17:16
@tofro: To be sure, despite the fact that the Standard explicitly describes the result of an equality comparison between a pointer which points just past the last element of one array and a pointer to the start of another array that happens to immediately follow it in memory (the pointers should be reported equal, and the comparison should presumably have no side effects) clang and gcc sometimes treat such equality comparisons as though they yield Undefined Behavior, but that should be recognized as a bug in clang and gcc, and not as a defect in the code they process incorrectly. – supercat Jun 22 '22 at 17:20
@tofro: For instance, code like int buf[n]; int *p; for (p = buf; p < buf + n; p++) { ... } is common and perfectly legal, where buf + n is a pointer just past its end. In fact, "pointer just past the end" is permitted precisely so that such code can be written. It is not so easy to write this kind of loop without using a pointer past the end, either explicitly or implicitly. – Nate Eldredge Jun 22 '22 at 18:23
It does mean that a typical compiler must arrange that no object ever goes to the very end of memory (or end of a segment, etc), because then buf + n would overflow and the comparison p < buf + n would come out wrong. But that's not hard to do, e.g. just treat the last byte of memory as unavailable. – Nate Eldredge Jun 22 '22 at 18:29
1

@supercat: In contrary to your claim that such a comparison should report that pointers are equal, the C++ Standard explicitly says the result is unspecified. For C, standard wording and compiler behavior and defect report appear to all be covered well here: https://pvs-studio.com/en/blog/posts/cpp/0576/ – Ben Voigt Jun 22 '22 at 19:29
@BenVoigt: For C, which is the language at issue, N1570 6.5.9 Equality operators, paragraph 9 says "Two pointers compare equal if and only if...or one is a pointer to one past the end of one array object and the other is a pointer to the start of a different array object that happens to immediately follow the first array object in the address space." Further, even in C++ the behavior would be specified as having the comparison either yield 0 (presumably with no side effects) or yield 1 (likewise), even if the choice could be made in Unspecified fashion, but both clang and gcc are prone... – supercat Jun 22 '22 at 19:37
...to yield behavior which is consistent neither with the comparison yielding 0, nor with its yielding 1. – supercat Jun 22 '22 at 19:38
@supercat: You didn't account for the defect report described at the linked page, even though I specifically mentioned it in my last comment :( The rule you cite for C is not normative since the resolution of DR 260. https://www.open-std.org/jtc1/sc22/wg14/www/docs/dr_260.htm – Ben Voigt Jun 22 '22 at 20:52
@BenVoigt: That DR says that knowledge that given e.g. int x[],y[];, an observation that x+10 == y does not imply that the expression x[10] may be used to access y[0], and I have no problem with that. I see nothing, however, saying that if the address of y is assigned to p, and p is compared to x+10, the behavior of such a comparison would not in all cases be specified as either yielding 0 with no side effects, or yielding 1 with no side effects. Both gcc and clang, however, would be prone to behave as though code which accesses *p were instead written to access x[10]... – supercat Jun 22 '22 at 21:04
...and would thus be incapable of accessing y[0] even if p had in fact been formed by taking y's address. – supercat Jun 22 '22 at 21:05
1

@tofro, do you have any evidence that this is the rationale for the particular return value chosen? Please provide your references. – Toby Speight Jun 24 '22 at 07:20
@Ben Voigt the c++ standard says no such thing (the part you reference talks about comparisons between such a pointer and a pointer to a different object, not within the same array). – Remember Monica Jul 18 '22 at 23:31
@RememberMonica: I was referencing a claim concerning pointers to two different array objects that lie adjacent in memory. – Ben Voigt Jul 19 '22 at 14:54

score 14 · Answer 3 · answered Jun 23 '22 at 08:22

In the original K&R version of C, functions "returning" void simply did not exist (more generally, void did not exist).

So they initially had to return something (even if it was never used), and once that was defined, you simply couldn't remove the return value as someone, somewhere may have used it. It was changed from char * to void * sometime in history, but that did not break compatibility.

So, if you had to return something, some possible choices could have been:

returning a flag indicating whether the operation was successful or not. But other than triggering a memory fault and hence halting execution, memcpy can hardly fail
returning the number of bytes copied
returning the address of the buffer, by analogy with other functions that do the same
returning the address of the end of the buffer

In all those cases, the return value would be either one of the inputs or something easily derived from them. The choice is probably arbitrary, and then going along the same lines as other functions is probably the most natural choice.

score 7 · Answer 4 · answered Jun 23 '22 at 17:12

The basis for ISO C in 1990 was ANSI C in 1989 ('C89').

There is an actual companion document for C89 called Rationale for American National Standard for Information Systems -- Programming Language -- C in which the X3J11 committee explains why they did what they did.

With respect to memcpy, that document says (4.11.1)

memcpy, memset, memcmp, and memchr have been adopted from several existing implementations. The general goal was to provide equivalent capabilities (...)

That's about it for the official Rationale. Anything more would be speculation by us.

But that's as I understand the committee's role in general - to ratify existing consensus rather than invent anything new.

(Sometimes invention was required to handle incompatible approaches: 4.11.2 does add further words about the addition of memmove, to resolve the issues around speed (memcpy) versus generality (memmove)).

score 3 · Answer 5 · answered Jun 23 '22 at 16:04

3

Like most standardization processes, the C standard documented commonalities of existing practice. So the core reason that the standard required memcpy to return its first argument is that pre-standardization C implementations did this, and pre-standardization C code made use of the fact that they did.

supercat's answer, which does not answer the question, has good speculation about the possible historical origin of the behavior, which might or might not be correct. If you want to research this in detail, there is source code for historical early versions of Unix available which might shed light on whether there was originally an intent to return the value or whether it just happened to be in the right register.

answered Jun 23 '22 at 16:04

R.. GitHub STOP HELPING ICE

581
3
14

1

memcpy just follows strcpy. I believe I read somewhere Ritchie discussing this as being historically related to B's representation of an array as having a separate pointer word, so you couldn't return a much-more-useful pointer to the array element after the copied data (to the 0 terminator in the case of strings). So that's why Unix did what it did, and (as you say) that became established practice and later standardized. – dave Jun 23 '22 at 17:23
@another-dave - that business about B's representation leading to the situation in C - I would never have guessed that, thank you. Especially because I've always wondered why it didn't return a pointer-to-after-the-copied-data (strcpy especially) so that you could (more) efficiently run one after the other ... – davidbak Jun 23 '22 at 17:55
I wish I could recall where I read it :-( – dave Jun 23 '22 at 18:07

score -4 · Answer 6 · answered Jun 23 '22 at 05:28

-4

c was a stack-oriented language. In a simple implementation, return value is reserved on the stack, the parameters are pushed on to the stack, and local variables are reserved on the stack, and when the function returns --- the parameters and local variables are still there.

By default, any implementation that takes the '1st parameter' position as the 'return value' position returns the first parameter as the return value.

So memcpy and similar functions returned the first parameter just by default, because it was a simple implementation.

answered Jun 23 '22 at 05:28

david

307
1
6

3

Even though arguments and local variables are often (but definitely not always) on the stack, I expect many (most?) implementation to use a register for the return value. See https://en.wikipedia.org/wiki/Calling_convention for a few examples. For those using registers both for arguments and return values (and the same ones) your argument may stand. – jcaron Jun 23 '22 at 08:45
@jcaron, that's why I wrote "was", and used the past tense in the final sentence. – david Jun 23 '22 at 09:27
2

@david But it wasn't so even on early implementations return value came in a register. – Raffzahn Jun 23 '22 at 12:34
Really? Where would the return value of a function like int GetSomeInt(); be stored? I'm pretty sure that the earliest C compiler I worked on (Aztec C on Z-80) returned values in the HL register way back in the early 80s. – Flydog57 Jun 23 '22 at 20:45
1

Putting the return value on the stack, rather than in some conventional register, would be highly unusual on the PDP-11 (which is what is meant by 'early C implementations', not microcomputer systems). To do that on the -11 requires at least popping the return link into a scratch register, pushing the function value, and then doing an indirect jump to the popped link; contrast this with writing the return value to a register and doing a simple subroutine return. – dave Jun 23 '22 at 23:51

In the ISO 1990 C standard library, what was the rationale for having memcpy return one of its inputs?

6 Answers6