Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [X..X.X..X..X.X...XX.XXXX.X..................X..........X.XX.] duration: 50.159949617s trace: https://jaeger.dev.storj.io/trace/4d6ec53c41a9b5cc FAIL (2025-12-26 15:35:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, Piece ID: 54YJLOKMXQWUAZTVTW55OWXDNZZMA63GJZYXYEQYYVSJTL57PRCA) context deadline exceeded; piecestore: (Node ID: 1wkujwGr7G6yGYFLiYcxfBnN1pmKY6mHXoAzZWAjGCCiWWZNdh, Piece ID: 5UR7MJZMHVCWCPJFDPW7EGLT3BDXICZMIM76YPVUA6JODJD45RYQ) context deadline exceeded; piecestore: (Node ID: 1CMsUV4eQnyd4FLQYLi5xdPDurkVaL2kyn8EW4VigZsh8sdTv6, Piece ID: VRHQMBNM7WYXL626TSIDL75ZTVYJGENSGDOQ6I2O35KGALGKKSVA) context deadline exceeded ======LOGS====== ODZ2M5DFJDIPA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.484026986 (504h4m44.355110031s) [noise=true , address=64.49.13.169:4709 , piece_id=OSMERMZAP3E7KW2CYMCYQ6JAZA3MT55MLEQ5IDDDWD6ICK5XFALQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.484477945 (504h4m44.35556099s) [noise=true , address=77.90.31.19:4866 , piece_id=F32RITTYFC6EO5E7NTOTMNLJ2VMEHN46YOXPCWVG25VDPIP4B75Q, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.485078588 (504h4m44.356161633s) [noise=true , address=45.91.116.4:4963 , piece_id=5LDE6TJMLXDAFPDJ7ZMKKTS6LS22VDZKW7MWMVEGA2DU37F2PDYQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.485615390 (504h4m44.356698435s) [noise=true , address=45.91.116.11:4121 , piece_id=5UR7MJZMHVCWCPJFDPW7EGLT3BDXICZMIM76YPVUA6JODJD45RYQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.485637873 (504h4m44.356720918s) [noise=true , address=145.79.227.44:4205 , piece_id=I5ISL5NWNUMQKLO75E7EJTHKN3R2NDZ5LDBH72DMO2PTQS67R2VA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.486193952 (504h4m44.357276987s) [noise=true , address=77.90.31.14:4670 , piece_id=DRBNQDD6O3CHX5OYAMYMKS7SRI5A4QOFVOAEKI3GNF3JWUKC55JA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.486453666 (504h4m44.357536701s) [noise=true , address=45.91.116.14:4201 , piece_id=IHFNLZRXMP3GFZP65PZSWG5GZ2ZFKTG2YVYC6GQ2BFEGQHS33HBA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.486754038 (504h4m44.357837083s) [noise=true , address=64.49.13.170:4725 , piece_id=3XFNXHSV6HMV5IDZWXZ5XH7RWRYS2UXATNGZWQNTVY57DBB23GXQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.487114945 (504h4m44.35819799s) [noise=true , address=145.79.227.29:4954 , piece_id=A6GARORLQWVZCU2GTFJ555RIXW7PCZB4I3CUJ6OXXCFIBJCJHBYQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.487331979 (504h4m44.358415024s) [noise=true , address=77.90.31.16:4762 , piece_id=QWSBMY4YF6FGVTC5QOM46SRCUKXWXQMXITY3ALFJKSZ2FCOEA2HQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.487769442 (504h4m44.358852487s) [noise=true , address=77.90.31.12:4612 , piece_id=VKWIGDKWG7OZKGFJSXNY7Q5HJTZHMMSXC3BJ7676TVXEFC6LKEFA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.487874922 (504h4m44.358957967s) [noise=true , address=77.90.31.15:4703 , piece_id=54YJLOKMXQWUAZTVTW55OWXDNZZMA63GJZYXYEQYYVSJTL57PRCA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.488454076 (504h4m44.359537111s) [noise=true , address=45.91.116.13:4174 , piece_id=VRHQMBNM7WYXL626TSIDL75ZTVYJGENSGDOQ6I2O35KGALGKKSVA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.491854641 (504h4m44.362937686s) [noise=true , address=64.49.13.26:4945 , piece_id=7ISUES6U4NH3NZU3GUOR2IMXLVRTSPUSWUZNC2QICKZ3MJELN4HA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:45.797674129 (504h4m45.668757174s) [upload=442344]: Next returned nil segment Dec 26 15:33:45.797692585 (504h4m45.66877562s) [upload=442344]: Waiting for error group managing segments... Dec 26 15:33:46.130577946 (504h4m46.001660981s) [noise=true , address=193.149.164.143:4702 , piece_id=ORKNZ2J4IE7SSNJGTYPD5V4VKDJGGKXVE7Z7BW6ODZ2M5DFJDIPA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.205726252 (504h4m46.076809297s) [noise=true , address=64.49.13.26:4945 , piece_id=7ISUES6U4NH3NZU3GUOR2IMXLVRTSPUSWUZNC2QICKZ3MJELN4HA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.308550998 (504h4m46.179634043s) [noise=true , address=64.49.13.170:4725 , piece_id=3XFNXHSV6HMV5IDZWXZ5XH7RWRYS2UXATNGZWQNTVY57DBB23GXQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.577575107 (504h4m46.448658152s) [noise=true , address=45.91.116.14:4201 , piece_id=IHFNLZRXMP3GFZP65PZSWG5GZ2ZFKTG2YVYC6GQ2BFEGQHS33HBA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.615742895 (504h4m46.48682593s) [noise=true , address=45.91.116.11:4121 , piece_id=5UR7MJZMHVCWCPJFDPW7EGLT3BDXICZMIM76YPVUA6JODJD45RYQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.616326656 (504h4m46.487409691s) [noise=true , address=45.91.116.13:4174 , piece_id=VRHQMBNM7WYXL626TSIDL75ZTVYJGENSGDOQ6I2O35KGALGKKSVA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.699340129 (504h4m46.570423174s) [noise=true , address=45.91.116.4:4963 , piece_id=5LDE6TJMLXDAFPDJ7ZMKKTS6LS22VDZKW7MWMVEGA2DU37F2PDYQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.708773672 (504h4m46.579856707s) [noise=true , address=145.79.227.29:4954 , piece_id=A6GARORLQWVZCU2GTFJ555RIXW7PCZB4I3CUJ6OXXCFIBJCJHBYQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.955154153 (504h4m46.826237198s) [noise=true , address=77.90.31.15:4703 , piece_id=54YJLOKMXQWUAZTVTW55OWXDNZZMA63GJZYXYEQYYVSJTL57PRCA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.142693852 (504h4m47.013776897s) [noise=true , address=77.90.31.19:4866 , piece_id=F32RITTYFC6EO5E7NTOTMNLJ2VMEHN46YOXPCWVG25VDPIP4B75Q, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.276150466 (504h4m47.147233511s) [noise=true , address=145.79.227.40:4053 , piece_id=2J2EY3X3WFXWWXZBHC6R2EN3SSOT35BJJULBJWLT577PJA6DY7CQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.304118842 (504h4m47.175201877s) [noise=true , address=145.79.227.44:4205 , piece_id=I5ISL5NWNUMQKLO75E7EJTHKN3R2NDZ5LDBH72DMO2PTQS67R2VA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.440569149 (504h4m47.311652194s) [noise=true , address=145.79.227.30:4796 , piece_id=MYHNLDAKFLPH2A7JEVQCAT7YQAO54XJ2SFD2MGQPPAFVIDVO47KA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.449458225 (504h4m47.32054128s) [noise=true , address=77.90.31.14:4670 , piece_id=DRBNQDD6O3CHX5OYAMYMKS7SRI5A4QOFVOAEKI3GNF3JWUKC55JA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.449479796 (504h4m47.320562841s) [seg_pos={0 0}, upload=442344]: Segment reached optimal threshold of 14 pieces. Dec 26 15:33:47.449793704 (504h4m47.320876739s) [noise=true , address=77.90.31.16:4762 , piece_id=QWSBMY4YF6FGVTC5QOM46SRCUKXWXQMXITY3ALFJKSZ2FCOEA2HQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: write tcp 188.166.225.170:53070->77.90.31.16:4762: use of closed network connection; write tcp 188.166.225.170:53070->77.90.31.16:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53070->77.90.31.16:4762: use of closed network connection Dec 26 15:33:47.449856813 (504h4m47.320939858s) [noise=true , address=77.90.31.16:4762 , piece_id=QWSBMY4YF6FGVTC5QOM46SRCUKXWXQMXITY3ALFJKSZ2FCOEA2HQ, seg_pos={0 0}, upload=442344]: Long tail cancellation due to optimal threshold Dec 26 15:33:47.449994956 (504h4m47.321078001s) [noise=true , address=64.49.13.169:4709 , piece_id=OSMERMZAP3E7KW2CYMCYQ6JAZA3MT55MLEQ5IDDDWD6ICK5XFALQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48594->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48594->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48594->64.49.13.169:4709: use of closed network connection Dec 26 15:33:47.450015105 (504h4m47.32109815s) [noise=true , address=64.49.13.169:4709 , piece_id=OSMERMZAP3E7KW2CYMCYQ6JAZA3MT55MLEQ5IDDDWD6ICK5XFALQ, seg_pos={0 0}, upload=442344]: Long tail cancellation due to optimal threshold Dec 26 15:33:47.450029161 (504h4m47.321112206s) [noise=true , address=77.90.31.11:4576 , piece_id=T5UX5RSTPFYOI6OS744JCEB4NNNBLYP24XCW7LOEYTDPTKHXAGAQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv): protocol: read tcp 188.166.225.170:37370->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:33:47.450036004 (504h4m47.321119039s) [noise=true , address=77.90.31.11:4576 , piece_id=T5UX5RSTPFYOI6OS744JCEB4NNNBLYP24XCW7LOEYTDPTKHXAGAQ, seg_pos={0 0}, upload=442344]: Long tail cancellation due to optimal threshold Dec 26 15:33:47.450179719 (504h4m47.321262764s) [noise=true , address=77.90.31.12:4612 , piece_id=VKWIGDKWG7OZKGFJSXNY7Q5HJTZHMMSXC3BJ7676TVXEFC6LKEFA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 26 15:33:47.450190198 (504h4m47.321273243s) [noise=true , address=77.90.31.12:4612 , piece_id=VKWIGDKWG7OZKGFJSXNY7Q5HJTZHMMSXC3BJ7676TVXEFC6LKEFA, seg_pos={0 0}, upload=442344]: Long tail cancellation due to optimal threshold Dec 26 15:33:47.450218202 (504h4m47.321301247s) [seg_pos={0 0}, upload=442344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:33:47.450321839 (504h4m47.321404884s) [upload=442344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:33:47.450328582 (504h4m47.321411637s) [upload=442344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:33:47.743361678 (504h4m47.614444733s) [upload=442344]: Done uploading segments... Dec 26 15:33:47.743387397 (504h4m47.614470442s) [upload=442344]: Upload finished. err: <nil> Duration: 50.159962974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [....XX..X.X.....X.....X.X............XXX....XX...X......X..X] duration: 30.003059451s trace: https://jaeger.dev.storj.io/trace/9be7c0c11f39bee FAIL (2025-12-26 15:34:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== oad Dec 26 15:34:24.826087459 (504h5m24.697170494s) [upload=442354]: Done starting upload Dec 26 15:34:24.826260930 (504h5m24.697343975s) [upload=442354]: Uploading segments... Dec 26 15:34:24.826352635 (504h5m24.69743568s) [upload=442354]: Got next segment. Inline: false Dec 26 15:34:24.826393142 (504h5m24.697476177s) [upload=442354]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:34:24.826398652 (504h5m24.697481697s) [upload=442354]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:34:25.069136767 (504h5m24.940219802s) [seg_pos={0 0}, upload=442354]: Begin upload segment... Dec 26 15:34:25.069197152 (504h5m24.940280197s) [seg_pos={0 0}, upload=442354]: Done begin upload segment. Dec 26 15:34:25.069206269 (504h5m24.940289305s) [noise=true , address=145.79.227.49:4338 , piece_id=OMPXSOKMSP3R5IS37BSOLQEYS7CIAU5JPXNLIYUCSA2CJNL2AA5Q, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.069280942 (504h5m24.940363978s) [noise=true , address=145.79.227.12:4767 , piece_id=AUAHK44K72XSUM4TAFLU3OI3XRPDR6VPXXO3MGGA2NJHNZENMQIA, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.069683138 (504h5m24.940766173s) [noise=true , address=145.79.227.34:4905 , piece_id=WLAM7GPPCLH5OHH73G6EVTY22FW33W5JF225NKEAFJYXZWCYTJMQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.070066848 (504h5m24.941149883s) [noise=true , address=145.79.227.42:4121 , piece_id=SA2SSBVL4UJTKT73Y4DOYMF2Q2M7VLZPGYOHFFQZAOPC4SKTXCYQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.070864417 (504h5m24.941947452s) [noise=true , address=45.91.116.11:4119 , piece_id=PBDVQSN5VICKOVOFXVBMLLQ7NGSP3ZEFX4WYLRHZJREABXANP3XQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.071153237 (504h5m24.942236272s) [noise=true , address=45.91.116.9:4262 , piece_id=JRJSX37GU6WCQRWQZ7ZMKJR547LZ62R7YBPZSD5H22DNU3YFOUMA, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.071418823 (504h5m24.942501858s) [noise=true , address=45.91.116.7:4012 , piece_id=4NCEE3XHZGR2YG2M6O6E2VNLFJANSWJ6NOYDKSRHA7BDQSU7A5ZA, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.071696111 (504h5m24.942779156s) [noise=true , address=145.79.227.37:4566 , piece_id=NMFBRVNHEBDBBZG6DDQEOOWPF22R3IEZU37RE6NNWWF6ZRR4AR5A, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.072014076 (504h5m24.943097121s) [noise=true , address=45.91.116.13:4153 , piece_id=S3ZQ6GQCMDW2RXDFQ3WUVS2HKAYVJR322NB6QVU46TYEWPJOJENQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.072901637 (504h5m24.943984672s) [noise=true , address=45.91.116.14:4189 , piece_id=VZHRPJDDV23UFDULZJK6LMB6K7KVGS6NELYBXYFYANERN4FUMRCQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.073032866 (504h5m24.944115911s) [noise=true , address=145.79.227.21:4475 , piece_id=IQFEFERWXBGHEHTYDQJZBVSMQDNSC62ZMIGSNFGJIDH3ADJ4QMYQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.074164171 (504h5m24.945247216s) [noise=true , address=145.79.227.7:4294 , piece_id=R3FNMICJODJ6BRU25W56S7FMW5NRHBEFQ4HQCLGOGPPGG4Y6FYQA, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.074528744 (504h5m24.94561178s) [noise=true , address=145.79.227.2:4114 , piece_id=J4VHYAVXBOYT2FJFVHMT77L5IRTJD62JHMTB6UPIPECQ3C3FNAMQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.075153324 (504h5m24.946236369s) [noise=true , address=145.79.227.52:4446 , piece_id=T32XKE7LNKTCVH5W6CQDHH57AG75VPB24ANBJIURHXEAMVZTKYZQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.076188485 (504h5m24.94727152s) [noise=true , address=145.79.227.16:4910 , piece_id=VDB3UWC7KP66HITDIHUPHLE6Y4Q26I4HS3WRVTJRTJLRIE2L7U5Q, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:25.077403609 (504h5m24.948486644s) [noise=true , address=145.79.227.24:4581 , piece_id=MMXOFNJLE2GEGENCKB5PCXJD3OXPOTJYCZJAGPFCEMXPJHNWNOKQ, seg_pos={0 0}, upload=442354]: Uploading piece... Dec 26 15:34:26.795877152 (504h5m26.666960197s) [upload=442354]: Next returned nil segment Dec 26 15:34:26.795898002 (504h5m26.666981047s) [upload=442354]: Waiting for error group managing segments... Dec 26 15:34:27.105765855 (504h5m26.97684889s) [noise=true , address=145.79.227.2:4114 , piece_id=J4VHYAVXBOYT2FJFVHMT77L5IRTJD62JHMTB6UPIPECQ3C3FNAMQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.108123354 (504h5m26.979206399s) [noise=true , address=145.79.227.16:4910 , piece_id=VDB3UWC7KP66HITDIHUPHLE6Y4Q26I4HS3WRVTJRTJLRIE2L7U5Q, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.109456793 (504h5m26.980539828s) [noise=true , address=145.79.227.12:4767 , piece_id=AUAHK44K72XSUM4TAFLU3OI3XRPDR6VPXXO3MGGA2NJHNZENMQIA, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.112117218 (504h5m26.983200253s) [noise=true , address=145.79.227.24:4581 , piece_id=MMXOFNJLE2GEGENCKB5PCXJD3OXPOTJYCZJAGPFCEMXPJHNWNOKQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.112432809 (504h5m26.983515844s) [noise=true , address=145.79.227.37:4566 , piece_id=NMFBRVNHEBDBBZG6DDQEOOWPF22R3IEZU37RE6NNWWF6ZRR4AR5A, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.112853281 (504h5m26.983936316s) [noise=true , address=145.79.227.52:4446 , piece_id=T32XKE7LNKTCVH5W6CQDHH57AG75VPB24ANBJIURHXEAMVZTKYZQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.128952075 (504h5m27.00003511s) [noise=true , address=145.79.227.49:4338 , piece_id=OMPXSOKMSP3R5IS37BSOLQEYS7CIAU5JPXNLIYUCSA2CJNL2AA5Q, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.131756325 (504h5m27.00283936s) [noise=true , address=145.79.227.7:4294 , piece_id=R3FNMICJODJ6BRU25W56S7FMW5NRHBEFQ4HQCLGOGPPGG4Y6FYQA, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.133890639 (504h5m27.004973684s) [noise=true , address=145.79.227.21:4475 , piece_id=IQFEFERWXBGHEHTYDQJZBVSMQDNSC62ZMIGSNFGJIDH3ADJ4QMYQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.138440283 (504h5m27.009523318s) [noise=true , address=145.79.227.42:4121 , piece_id=SA2SSBVL4UJTKT73Y4DOYMF2Q2M7VLZPGYOHFFQZAOPC4SKTXCYQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.397490890 (504h5m27.268573935s) [noise=true , address=45.91.116.14:4189 , piece_id=VZHRPJDDV23UFDULZJK6LMB6K7KVGS6NELYBXYFYANERN4FUMRCQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.459581161 (504h5m27.330664206s) [noise=true , address=45.91.116.13:4153 , piece_id=S3ZQ6GQCMDW2RXDFQ3WUVS2HKAYVJR322NB6QVU46TYEWPJOJENQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.729226993 (504h5m27.600310038s) [noise=true , address=45.91.116.9:4262 , piece_id=JRJSX37GU6WCQRWQZ7ZMKJR547LZ62R7YBPZSD5H22DNU3YFOUMA, seg_pos={0 0}, upload=442354]: Done uploading piece. err: <nil> Dec 26 15:34:27.729249386 (504h5m27.600332421s) [seg_pos={0 0}, upload=442354]: Segment reached optimal threshold of 13 pieces. Dec 26 15:34:27.729537935 (504h5m27.600620981s) [noise=true , address=45.91.116.7:4012 , piece_id=4NCEE3XHZGR2YG2M6O6E2VNLFJANSWJ6NOYDKSRHA7BDQSU7A5ZA, seg_pos={0 0}, upload=442354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:54992->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:54992->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54992->45.91.116.7:4012: use of closed network connection Dec 26 15:34:27.729547844 (504h5m27.600630899s) [noise=true , address=45.91.116.7:4012 , piece_id=4NCEE3XHZGR2YG2M6O6E2VNLFJANSWJ6NOYDKSRHA7BDQSU7A5ZA, seg_pos={0 0}, upload=442354]: Long tail cancellation due to optimal threshold Dec 26 15:34:27.729589394 (504h5m27.600672439s) [noise=true , address=145.79.227.34:4905 , piece_id=WLAM7GPPCLH5OHH73G6EVTY22FW33W5JF225NKEAFJYXZWCYTJMQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 26 15:34:27.729597139 (504h5m27.600680174s) [noise=true , address=145.79.227.34:4905 , piece_id=WLAM7GPPCLH5OHH73G6EVTY22FW33W5JF225NKEAFJYXZWCYTJMQ, seg_pos={0 0}, upload=442354]: Long tail cancellation due to optimal threshold Dec 26 15:34:27.729978565 (504h5m27.60106162s) [noise=true , address=45.91.116.11:4119 , piece_id=PBDVQSN5VICKOVOFXVBMLLQ7NGSP3ZEFX4WYLRHZJREABXANP3XQ, seg_pos={0 0}, upload=442354]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: write tcp 188.166.225.170:50458->45.91.116.11:4119: use of closed network connection; write tcp 188.166.225.170:50458->45.91.116.11:4119: use of closed network connection Dec 26 15:34:27.730003902 (504h5m27.601086947s) [noise=true , address=45.91.116.11:4119 , piece_id=PBDVQSN5VICKOVOFXVBMLLQ7NGSP3ZEFX4WYLRHZJREABXANP3XQ, seg_pos={0 0}, upload=442354]: Long tail cancellation due to optimal threshold Dec 26 15:34:27.730035853 (504h5m27.601118898s) [seg_pos={0 0}, upload=442354]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:34:27.730130644 (504h5m27.601213689s) [upload=442354]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:34:27.730136014 (504h5m27.601219049s) [upload=442354]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:34:28.030174941 (504h5m27.901257976s) [upload=442354]: Done uploading segments... Dec 26 15:34:28.030196833 (504h5m27.901279868s) [upload=442354]: Upload finished. err: <nil> Duration: 30.003075461s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [.XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X.....X.X.] duration: 3.405466032s trace: https://jaeger.dev.storj.io/trace/71972eeda6034f51 FAIL (2025-12-26 15:34:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.405493073s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX..X.X..X..X.X...XX.XXXX.X..................X..........X.XX] duration: 1m0.002745529s trace: https://jaeger.dev.storj.io/trace/4d6ec53c41a9b5cc FAIL (2025-12-26 15:34:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1hMDD5K7WDjvN4ZgcpwpnPFb6X1ncCfyucfFrpTUshd8Ye1B6D, Piece ID: 54YJLOKMXQWUAZTVTW55OWXDNZZMA63GJZYXYEQYYVSJTL57PRCA) context deadline exceeded; piecestore: (Node ID: 1wkujwGr7G6yGYFLiYcxfBnN1pmKY6mHXoAzZWAjGCCiWWZNdh, Piece ID: 5UR7MJZMHVCWCPJFDPW7EGLT3BDXICZMIM76YPVUA6JODJD45RYQ) context deadline exceeded; piecestore: (Node ID: 1CMsUV4eQnyd4FLQYLi5xdPDurkVaL2kyn8EW4VigZsh8sdTv6, Piece ID: VRHQMBNM7WYXL626TSIDL75ZTVYJGENSGDOQ6I2O35KGALGKKSVA) context deadline exceeded ======LOGS====== ODZ2M5DFJDIPA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.484026986 (504h4m44.355110031s) [noise=true , address=64.49.13.169:4709 , piece_id=OSMERMZAP3E7KW2CYMCYQ6JAZA3MT55MLEQ5IDDDWD6ICK5XFALQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.484477945 (504h4m44.35556099s) [noise=true , address=77.90.31.19:4866 , piece_id=F32RITTYFC6EO5E7NTOTMNLJ2VMEHN46YOXPCWVG25VDPIP4B75Q, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.485078588 (504h4m44.356161633s) [noise=true , address=45.91.116.4:4963 , piece_id=5LDE6TJMLXDAFPDJ7ZMKKTS6LS22VDZKW7MWMVEGA2DU37F2PDYQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.485615390 (504h4m44.356698435s) [noise=true , address=45.91.116.11:4121 , piece_id=5UR7MJZMHVCWCPJFDPW7EGLT3BDXICZMIM76YPVUA6JODJD45RYQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.485637873 (504h4m44.356720918s) [noise=true , address=145.79.227.44:4205 , piece_id=I5ISL5NWNUMQKLO75E7EJTHKN3R2NDZ5LDBH72DMO2PTQS67R2VA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.486193952 (504h4m44.357276987s) [noise=true , address=77.90.31.14:4670 , piece_id=DRBNQDD6O3CHX5OYAMYMKS7SRI5A4QOFVOAEKI3GNF3JWUKC55JA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.486453666 (504h4m44.357536701s) [noise=true , address=45.91.116.14:4201 , piece_id=IHFNLZRXMP3GFZP65PZSWG5GZ2ZFKTG2YVYC6GQ2BFEGQHS33HBA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.486754038 (504h4m44.357837083s) [noise=true , address=64.49.13.170:4725 , piece_id=3XFNXHSV6HMV5IDZWXZ5XH7RWRYS2UXATNGZWQNTVY57DBB23GXQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.487114945 (504h4m44.35819799s) [noise=true , address=145.79.227.29:4954 , piece_id=A6GARORLQWVZCU2GTFJ555RIXW7PCZB4I3CUJ6OXXCFIBJCJHBYQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.487331979 (504h4m44.358415024s) [noise=true , address=77.90.31.16:4762 , piece_id=QWSBMY4YF6FGVTC5QOM46SRCUKXWXQMXITY3ALFJKSZ2FCOEA2HQ, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.487769442 (504h4m44.358852487s) [noise=true , address=77.90.31.12:4612 , piece_id=VKWIGDKWG7OZKGFJSXNY7Q5HJTZHMMSXC3BJ7676TVXEFC6LKEFA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.487874922 (504h4m44.358957967s) [noise=true , address=77.90.31.15:4703 , piece_id=54YJLOKMXQWUAZTVTW55OWXDNZZMA63GJZYXYEQYYVSJTL57PRCA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.488454076 (504h4m44.359537111s) [noise=true , address=45.91.116.13:4174 , piece_id=VRHQMBNM7WYXL626TSIDL75ZTVYJGENSGDOQ6I2O35KGALGKKSVA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:44.491854641 (504h4m44.362937686s) [noise=true , address=64.49.13.26:4945 , piece_id=7ISUES6U4NH3NZU3GUOR2IMXLVRTSPUSWUZNC2QICKZ3MJELN4HA, seg_pos={0 0}, upload=442344]: Uploading piece... Dec 26 15:33:45.797674129 (504h4m45.668757174s) [upload=442344]: Next returned nil segment Dec 26 15:33:45.797692585 (504h4m45.66877562s) [upload=442344]: Waiting for error group managing segments... Dec 26 15:33:46.130577946 (504h4m46.001660981s) [noise=true , address=193.149.164.143:4702 , piece_id=ORKNZ2J4IE7SSNJGTYPD5V4VKDJGGKXVE7Z7BW6ODZ2M5DFJDIPA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.205726252 (504h4m46.076809297s) [noise=true , address=64.49.13.26:4945 , piece_id=7ISUES6U4NH3NZU3GUOR2IMXLVRTSPUSWUZNC2QICKZ3MJELN4HA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.308550998 (504h4m46.179634043s) [noise=true , address=64.49.13.170:4725 , piece_id=3XFNXHSV6HMV5IDZWXZ5XH7RWRYS2UXATNGZWQNTVY57DBB23GXQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.577575107 (504h4m46.448658152s) [noise=true , address=45.91.116.14:4201 , piece_id=IHFNLZRXMP3GFZP65PZSWG5GZ2ZFKTG2YVYC6GQ2BFEGQHS33HBA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.615742895 (504h4m46.48682593s) [noise=true , address=45.91.116.11:4121 , piece_id=5UR7MJZMHVCWCPJFDPW7EGLT3BDXICZMIM76YPVUA6JODJD45RYQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.616326656 (504h4m46.487409691s) [noise=true , address=45.91.116.13:4174 , piece_id=VRHQMBNM7WYXL626TSIDL75ZTVYJGENSGDOQ6I2O35KGALGKKSVA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.699340129 (504h4m46.570423174s) [noise=true , address=45.91.116.4:4963 , piece_id=5LDE6TJMLXDAFPDJ7ZMKKTS6LS22VDZKW7MWMVEGA2DU37F2PDYQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.708773672 (504h4m46.579856707s) [noise=true , address=145.79.227.29:4954 , piece_id=A6GARORLQWVZCU2GTFJ555RIXW7PCZB4I3CUJ6OXXCFIBJCJHBYQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:46.955154153 (504h4m46.826237198s) [noise=true , address=77.90.31.15:4703 , piece_id=54YJLOKMXQWUAZTVTW55OWXDNZZMA63GJZYXYEQYYVSJTL57PRCA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.142693852 (504h4m47.013776897s) [noise=true , address=77.90.31.19:4866 , piece_id=F32RITTYFC6EO5E7NTOTMNLJ2VMEHN46YOXPCWVG25VDPIP4B75Q, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.276150466 (504h4m47.147233511s) [noise=true , address=145.79.227.40:4053 , piece_id=2J2EY3X3WFXWWXZBHC6R2EN3SSOT35BJJULBJWLT577PJA6DY7CQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.304118842 (504h4m47.175201877s) [noise=true , address=145.79.227.44:4205 , piece_id=I5ISL5NWNUMQKLO75E7EJTHKN3R2NDZ5LDBH72DMO2PTQS67R2VA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.440569149 (504h4m47.311652194s) [noise=true , address=145.79.227.30:4796 , piece_id=MYHNLDAKFLPH2A7JEVQCAT7YQAO54XJ2SFD2MGQPPAFVIDVO47KA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.449458225 (504h4m47.32054128s) [noise=true , address=77.90.31.14:4670 , piece_id=DRBNQDD6O3CHX5OYAMYMKS7SRI5A4QOFVOAEKI3GNF3JWUKC55JA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: <nil> Dec 26 15:33:47.449479796 (504h4m47.320562841s) [seg_pos={0 0}, upload=442344]: Segment reached optimal threshold of 14 pieces. Dec 26 15:33:47.449793704 (504h4m47.320876739s) [noise=true , address=77.90.31.16:4762 , piece_id=QWSBMY4YF6FGVTC5QOM46SRCUKXWXQMXITY3ALFJKSZ2FCOEA2HQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: write tcp 188.166.225.170:53070->77.90.31.16:4762: use of closed network connection; write tcp 188.166.225.170:53070->77.90.31.16:4762: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53070->77.90.31.16:4762: use of closed network connection Dec 26 15:33:47.449856813 (504h4m47.320939858s) [noise=true , address=77.90.31.16:4762 , piece_id=QWSBMY4YF6FGVTC5QOM46SRCUKXWXQMXITY3ALFJKSZ2FCOEA2HQ, seg_pos={0 0}, upload=442344]: Long tail cancellation due to optimal threshold Dec 26 15:33:47.449994956 (504h4m47.321078001s) [noise=true , address=64.49.13.169:4709 , piece_id=OSMERMZAP3E7KW2CYMCYQ6JAZA3MT55MLEQ5IDDDWD6ICK5XFALQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:48594->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:48594->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48594->64.49.13.169:4709: use of closed network connection Dec 26 15:33:47.450015105 (504h4m47.32109815s) [noise=true , address=64.49.13.169:4709 , piece_id=OSMERMZAP3E7KW2CYMCYQ6JAZA3MT55MLEQ5IDDDWD6ICK5XFALQ, seg_pos={0 0}, upload=442344]: Long tail cancellation due to optimal threshold Dec 26 15:33:47.450029161 (504h4m47.321112206s) [noise=true , address=77.90.31.11:4576 , piece_id=T5UX5RSTPFYOI6OS744JCEB4NNNBLYP24XCW7LOEYTDPTKHXAGAQ, seg_pos={0 0}, upload=442344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1br8Q73A3rDz8b6fZxop71zh5vdEcdye9D7ta8aBpsctwLBfAv): protocol: read tcp 188.166.225.170:37370->77.90.31.11:4576: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:33:47.450036004 (504h4m47.321119039s) [noise=true , address=77.90.31.11:4576 , piece_id=T5UX5RSTPFYOI6OS744JCEB4NNNBLYP24XCW7LOEYTDPTKHXAGAQ, seg_pos={0 0}, upload=442344]: Long tail cancellation due to optimal threshold Dec 26 15:33:47.450179719 (504h4m47.321262764s) [noise=true , address=77.90.31.12:4612 , piece_id=VKWIGDKWG7OZKGFJSXNY7Q5HJTZHMMSXC3BJ7676TVXEFC6LKEFA, seg_pos={0 0}, upload=442344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 26 15:33:47.450190198 (504h4m47.321273243s) [noise=true , address=77.90.31.12:4612 , piece_id=VKWIGDKWG7OZKGFJSXNY7Q5HJTZHMMSXC3BJ7676TVXEFC6LKEFA, seg_pos={0 0}, upload=442344]: Long tail cancellation due to optimal threshold Dec 26 15:33:47.450218202 (504h4m47.321301247s) [seg_pos={0 0}, upload=442344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:33:47.450321839 (504h4m47.321404884s) [upload=442344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:33:47.450328582 (504h4m47.321411637s) [upload=442344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:33:47.743361678 (504h4m47.614444733s) [upload=442344]: Done uploading segments... Dec 26 15:33:47.743387397 (504h4m47.614470442s) [upload=442344]: Upload finished. err: <nil> Duration: 1m0.00276163s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.....XX..X.X.....X.....X.X............XXX....XX...X......X..] duration: 7.011270652s trace: https://jaeger.dev.storj.io/trace/db5d56fbf5ab72f FAIL (2025-12-26 15:34:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 327]: Starting upload Dec 26 15:32:27.150869037 (504h3m27.021952082s) [upload=442327]: Done starting upload Dec 26 15:32:27.150966753 (504h3m27.022049798s) [upload=442327]: Uploading segments... Dec 26 15:32:27.151111518 (504h3m27.022194563s) [upload=442327]: Got next segment. Inline: false Dec 26 15:32:27.151160903 (504h3m27.022243938s) [upload=442327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:32:27.151170230 (504h3m27.022253265s) [upload=442327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:32:27.396820902 (504h3m27.267903947s) [seg_pos={0 0}, upload=442327]: Begin upload segment... Dec 26 15:32:27.396878382 (504h3m27.267961427s) [seg_pos={0 0}, upload=442327]: Done begin upload segment. Dec 26 15:32:27.396911815 (504h3m27.267994851s) [noise=true , address=45.91.116.7:4010 , piece_id=BIKQMUUUFMHSCKHXV3J2U4JRQKN67YM4VCOKYGRTP2GJKF57OUCA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.397064496 (504h3m27.268147531s) [noise=true , address=145.79.227.14:4837 , piece_id=7YCJGNJIGCM73ELVSK4VXO4HMFLHATCDIRZCCONCHZHXZIAZKD5Q, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.397617349 (504h3m27.268700414s) [noise=true , address=145.79.227.37:4567 , piece_id=4I2XA5ZFZ7MLVQN2JDLOWTKGXLSACCFJWPY7CTEHSXK424H3F6OA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398128021 (504h3m27.269211056s) [noise=true , address=145.79.227.15:4873 , piece_id=P6PRZN3BU7ZQITRIP2CWXP3MXAT35YUQEQHBA6MZCH3CGSYW6PXQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398146155 (504h3m27.26922919s) [noise=true , address=145.79.227.41:4085 , piece_id=5F3L4TMU2ZUHQHVNYZNEGNZEAJEHX6NWEK2OUXP6CICSGAEVNTMA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398458009 (504h3m27.269541044s) [noise=true , address=145.79.227.12:4767 , piece_id=TWYZFE7ANYIMNH3HF4E6OQDADJRM2OVJOKRQS6ETI36VKRRUM57A, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398467878 (504h3m27.269550923s) [noise=true , address=145.79.227.42:4123 , piece_id=PFMH6NY3GNEY5ZNMWMHAOXOQLFTZGLWNJPQNDUGHHOVTT3OI4PBQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398732943 (504h3m27.269815978s) [noise=true , address=145.79.227.50:4373 , piece_id=KB6YTMU5L7Z45LPRIZLEKBXTGBJM2AP74T3MYATZ6OY62J2IRAXA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398758772 (504h3m27.269841807s) [noise=true , address=45.91.116.14:4192 , piece_id=C22A5VXQ5CGXRP6SUFAJJTKYJQU6KRDSWTABIQGTDQ2JBG33GNPA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399028857 (504h3m27.270111902s) [noise=true , address=77.90.31.11:4555 , piece_id=SKCTG3FCS24RPX7RHIOHRKE77IA7WVLLLHAUJPWAH2CRX4227GEQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399043615 (504h3m27.27012666s) [noise=true , address=145.79.227.45:4231 , piece_id=YKOCOIXMMBVHHNICPFVIRPMEQR35M5TDALKKV4DUQSYQBEHI4AUQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399316584 (504h3m27.27039963s) [noise=true , address=145.79.227.52:4445 , piece_id=3NYGEGKMMV4M5WOK65YK7PGVONTEYD3MP7WP5TR2RRIZBREESTSA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399323958 (504h3m27.270407003s) [noise=true , address=45.91.116.11:4117 , piece_id=ZMEIAXIZPJ4OUBYYOTARZUYYZVREXS6S6I5LPKZEEEF23A6CQBWQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399593432 (504h3m27.270676467s) [noise=true , address=145.79.227.7:4294 , piece_id=G4FDHQR4EY22BSGCFMUW4PC2DVWY2H3EPPGPISZSF5TJFQBBT2EA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399869507 (504h3m27.270952543s) [noise=true , address=145.79.227.24:4581 , piece_id=6RP6INDHP4HNZBINR4JTSUU2WU2RV6NIOPXYIB564HCJCPHLB7WQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.400755334 (504h3m27.271838369s) [noise=true , address=77.90.31.14:4664 , piece_id=AQMWBOSHLHIP3R54PH2GULF5NPUR5VNB75SCWNMFQCGLSBZ7DHQQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:29.102300563 (504h3m28.973383608s) [upload=442327]: Next returned nil segment Dec 26 15:32:29.102325200 (504h3m28.973408235s) [upload=442327]: Waiting for error group managing segments... Dec 26 15:32:29.388594837 (504h3m29.259677872s) [noise=true , address=145.79.227.7:4294 , piece_id=G4FDHQR4EY22BSGCFMUW4PC2DVWY2H3EPPGPISZSF5TJFQBBT2EA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.389572970 (504h3m29.260656015s) [noise=true , address=145.79.227.50:4373 , piece_id=KB6YTMU5L7Z45LPRIZLEKBXTGBJM2AP74T3MYATZ6OY62J2IRAXA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.400604706 (504h3m29.271687741s) [noise=true , address=145.79.227.15:4873 , piece_id=P6PRZN3BU7ZQITRIP2CWXP3MXAT35YUQEQHBA6MZCH3CGSYW6PXQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.403736069 (504h3m29.274819114s) [noise=true , address=145.79.227.14:4837 , piece_id=7YCJGNJIGCM73ELVSK4VXO4HMFLHATCDIRZCCONCHZHXZIAZKD5Q, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.404653175 (504h3m29.27573621s) [noise=true , address=145.79.227.24:4581 , piece_id=6RP6INDHP4HNZBINR4JTSUU2WU2RV6NIOPXYIB564HCJCPHLB7WQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.407490368 (504h3m29.278573403s) [noise=true , address=145.79.227.42:4123 , piece_id=PFMH6NY3GNEY5ZNMWMHAOXOQLFTZGLWNJPQNDUGHHOVTT3OI4PBQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.408188768 (504h3m29.279271803s) [noise=true , address=145.79.227.41:4085 , piece_id=5F3L4TMU2ZUHQHVNYZNEGNZEAJEHX6NWEK2OUXP6CICSGAEVNTMA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.410656246 (504h3m29.281739281s) [noise=true , address=145.79.227.52:4445 , piece_id=3NYGEGKMMV4M5WOK65YK7PGVONTEYD3MP7WP5TR2RRIZBREESTSA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.410946158 (504h3m29.282029193s) [noise=true , address=145.79.227.37:4567 , piece_id=4I2XA5ZFZ7MLVQN2JDLOWTKGXLSACCFJWPY7CTEHSXK424H3F6OA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.413745288 (504h3m29.284828334s) [noise=true , address=145.79.227.45:4231 , piece_id=YKOCOIXMMBVHHNICPFVIRPMEQR35M5TDALKKV4DUQSYQBEHI4AUQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.425009076 (504h3m29.296092111s) [noise=true , address=145.79.227.12:4767 , piece_id=TWYZFE7ANYIMNH3HF4E6OQDADJRM2OVJOKRQS6ETI36VKRRUM57A, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.670341572 (504h3m29.541424607s) [noise=true , address=45.91.116.11:4117 , piece_id=ZMEIAXIZPJ4OUBYYOTARZUYYZVREXS6S6I5LPKZEEEF23A6CQBWQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.713495595 (504h3m29.58457863s) [noise=true , address=45.91.116.7:4010 , piece_id=BIKQMUUUFMHSCKHXV3J2U4JRQKN67YM4VCOKYGRTP2GJKF57OUCA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.713513468 (504h3m29.584596514s) [seg_pos={0 0}, upload=442327]: Segment reached optimal threshold of 13 pieces. Dec 26 15:32:29.713767834 (504h3m29.584850879s) [noise=true , address=45.91.116.14:4192 , piece_id=C22A5VXQ5CGXRP6SUFAJJTKYJQU6KRDSWTABIQGTDQ2JBG33GNPA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:55362->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:55362->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55362->45.91.116.14:4192: use of closed network connection Dec 26 15:32:29.713777161 (504h3m29.584860206s) [noise=true , address=45.91.116.14:4192 , piece_id=C22A5VXQ5CGXRP6SUFAJJTKYJQU6KRDSWTABIQGTDQ2JBG33GNPA, seg_pos={0 0}, upload=442327]: Long tail cancellation due to optimal threshold Dec 26 15:32:29.714024021 (504h3m29.585107066s) [noise=true , address=77.90.31.11:4555 , piece_id=SKCTG3FCS24RPX7RHIOHRKE77IA7WVLLLHAUJPWAH2CRX4227GEQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: read tcp 188.166.225.170:38774->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:32:29.714034621 (504h3m29.585117666s) [noise=true , address=77.90.31.11:4555 , piece_id=SKCTG3FCS24RPX7RHIOHRKE77IA7WVLLLHAUJPWAH2CRX4227GEQ, seg_pos={0 0}, upload=442327]: Long tail cancellation due to optimal threshold Dec 26 15:32:29.714088314 (504h3m29.585171359s) [noise=true , address=77.90.31.14:4664 , piece_id=AQMWBOSHLHIP3R54PH2GULF5NPUR5VNB75SCWNMFQCGLSBZ7DHQQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 26 15:32:29.714097962 (504h3m29.585181007s) [noise=true , address=77.90.31.14:4664 , piece_id=AQMWBOSHLHIP3R54PH2GULF5NPUR5VNB75SCWNMFQCGLSBZ7DHQQ, seg_pos={0 0}, upload=442327]: Long tail cancellation due to optimal threshold Dec 26 15:32:29.714116087 (504h3m29.585199122s) [seg_pos={0 0}, upload=442327]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:32:29.714198333 (504h3m29.585281408s) [upload=442327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:32:29.714207089 (504h3m29.585290124s) [upload=442327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:32:30.011069438 (504h3m29.882152473s) [upload=442327]: Done uploading segments... Dec 26 15:32:30.011090958 (504h3m29.882174003s) [upload=442327]: Upload finished. err: <nil> Duration: 7.011403918s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XX..X.X..X..X.X...XX.XXXX.X..................X..........X.X] duration: 1m0.001729695s trace: https://jaeger.dev.storj.io/trace/61dc4d67ec619425 FAIL (2025-12-26 15:33:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12bFT8wMeMrsm5byCeSjJQQUXCWmDtyBqiiu1KqpXMoxTHAakV9, Piece ID: XKARWE73SY3O2NEUUERD5IJ2RJ3LNNNTQERYZOERRB42RNVOB4EA) context deadline exceeded; piecestore: (Node ID: 1pX3YNcCwaUeMstL4NNwvnKQmHbmdxLTUc8sDHwyvZGZtzmYgW, Piece ID: DAJ7JX7MMPCOOVWLZMKSTJJCGZEVEQWXRUSKKXDLE6Q4WYDGDVYA) context deadline exceeded; piecestore: (Node ID: 122zPTREDCWCmywLC5fsUzDr59NHxzFMAUTp23icLjJzL7mHgdm, Piece ID: NBLKWG5PLHW2SQ563RBNNLIEH6ZBNLZRHPYPPFH26XK6W5DHXVQQ) context deadline exceeded ======LOGS====== 6 15:32:44.260628716 (504h3m44.131711761s) [noise=true , address=145.79.227.47:4281 , piece_id=LE7IK27V5WYDLT6B6PZOLOLO6U37SZPXPLDDD6TO7LFSS7IPC65Q, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.260940239 (504h3m44.132023284s) [noise=true , address=45.91.116.4:4960 , piece_id=TCCJBCOL23W7BNAPUQ3K6G5VR6EP2KRHDVWRZJOOAK2GX36IIWMA, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.261208099 (504h3m44.132291134s) [noise=true , address=145.79.227.34:4927 , piece_id=MHQ5HOSRWVH4MN5PQC6ABC2XZWZ3BMAJ6BOW4PVEZ352X3IJR3YQ, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.261713852 (504h3m44.132796897s) [noise=true , address=145.79.227.44:4227 , piece_id=PMANVYTX3G3AQHIJSXSAYRI4FSF56JUT5BQK6QZZKPFT2Z6PX4TA, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.262188957 (504h3m44.133272002s) [noise=true , address=45.91.116.10:4097 , piece_id=HOWSTA5KFZ7NGRXZPDDSZD2BFFFDZTJUTJL3GEWZ5GHCKB2AUTFQ, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.263588081 (504h3m44.134671126s) [noise=true , address=77.90.31.19:4873 , piece_id=GXZZHUADT4462UKIFXSEZMVIUJ32J4CTQIQAJYI5ATJJM6AEF5WA, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.264038048 (504h3m44.135121093s) [noise=true , address=145.79.227.30:4785 , piece_id=EVFF6J2FLJNH6YGLWDGJCPCSJTLWM473ZJVLJDTT4OF4XYGGQPUQ, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.264488996 (504h3m44.135572041s) [noise=true , address=77.90.31.11:4574 , piece_id=DAJ7JX7MMPCOOVWLZMKSTJJCGZEVEQWXRUSKKXDLE6Q4WYDGDVYA, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.266184726 (504h3m44.137267771s) [noise=true , address=77.90.31.21:4908 , piece_id=4WCHAGG4O2VVVXZZSJC2H5XQADPVZHQJHUQYPMAI4GI2Z3CWE7HQ, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.267910661 (504h3m44.138993707s) [noise=true , address=145.79.227.40:4059 , piece_id=WOW4SDDHF2FV3472XE6LLFWJOL2LCVGWTYOJ6ZECDY5OQK6LXGTA, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.269608244 (504h3m44.140691289s) [noise=true , address=77.90.31.10:4529 , piece_id=NBLKWG5PLHW2SQ563RBNNLIEH6ZBNLZRHPYPPFH26XK6W5DHXVQQ, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.270943066 (504h3m44.142026101s) [noise=true , address=77.90.31.18:4820 , piece_id=XKARWE73SY3O2NEUUERD5IJ2RJ3LNNNTQERYZOERRB42RNVOB4EA, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.272213135 (504h3m44.14329617s) [noise=true , address=64.49.13.169:4706 , piece_id=PY4667A2DCONGTD2CGFZ6I3DZON3JGBPEDAFRECAWGPND7S2P45A, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.273524422 (504h3m44.144607467s) [noise=true , address=77.90.31.17:4792 , piece_id=SEZ64NSDSE7PFCGOWHVZSPJHYOXK4AGUJTTO6YBYXMYPSZH2PBOQ, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.274861838 (504h3m44.145944873s) [noise=true , address=77.90.31.16:4768 , piece_id=KAIFWFKT36FG2672SRWVHB3KF3S2X2JTXJCIAMW5VBMHZUVUDFRQ, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:44.275434809 (504h3m44.146517854s) [noise=true , address=145.79.227.43:4175 , piece_id=TDHCMWDBTBTJQGDW3UDSIAEZPAHDCGUNOALAAJUXM2DY7KYKIJIQ, seg_pos={0 0}, upload=442332]: Uploading piece... Dec 26 15:32:45.735001559 (504h3m45.606084594s) [upload=442332]: Next returned nil segment Dec 26 15:32:45.735015345 (504h3m45.60609839s) [upload=442332]: Waiting for error group managing segments... Dec 26 15:32:46.157631319 (504h3m46.028714364s) [noise=true , address=145.79.227.47:4281 , piece_id=LE7IK27V5WYDLT6B6PZOLOLO6U37SZPXPLDDD6TO7LFSS7IPC65Q, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:46.164758311 (504h3m46.035841346s) [noise=true , address=145.79.227.43:4175 , piece_id=TDHCMWDBTBTJQGDW3UDSIAEZPAHDCGUNOALAAJUXM2DY7KYKIJIQ, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:46.320405691 (504h3m46.191488726s) [noise=true , address=77.90.31.21:4908 , piece_id=4WCHAGG4O2VVVXZZSJC2H5XQADPVZHQJHUQYPMAI4GI2Z3CWE7HQ, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:46.359214530 (504h3m46.230297575s) [noise=true , address=77.90.31.11:4574 , piece_id=DAJ7JX7MMPCOOVWLZMKSTJJCGZEVEQWXRUSKKXDLE6Q4WYDGDVYA, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:46.396758129 (504h3m46.267841164s) [noise=true , address=45.91.116.4:4960 , piece_id=TCCJBCOL23W7BNAPUQ3K6G5VR6EP2KRHDVWRZJOOAK2GX36IIWMA, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:46.438396866 (504h3m46.309479911s) [noise=true , address=77.90.31.16:4768 , piece_id=KAIFWFKT36FG2672SRWVHB3KF3S2X2JTXJCIAMW5VBMHZUVUDFRQ, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:46.483980976 (504h3m46.355064021s) [noise=true , address=77.90.31.19:4873 , piece_id=GXZZHUADT4462UKIFXSEZMVIUJ32J4CTQIQAJYI5ATJJM6AEF5WA, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:46.494760142 (504h3m46.365843187s) [noise=true , address=77.90.31.10:4529 , piece_id=NBLKWG5PLHW2SQ563RBNNLIEH6ZBNLZRHPYPPFH26XK6W5DHXVQQ, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:46.741890932 (504h3m46.612973967s) [noise=true , address=77.90.31.18:4820 , piece_id=XKARWE73SY3O2NEUUERD5IJ2RJ3LNNNTQERYZOERRB42RNVOB4EA, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:47.000445775 (504h3m46.87152883s) [noise=true , address=77.90.31.13:4625 , piece_id=I7KZWLCK7GLZYN6JIBIH4BY3OW4HXR7HH6YLMUQWP5LELNXEAQAA, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:47.092510673 (504h3m46.963593708s) [noise=true , address=145.79.227.44:4227 , piece_id=PMANVYTX3G3AQHIJSXSAYRI4FSF56JUT5BQK6QZZKPFT2Z6PX4TA, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:47.096262397 (504h3m46.967345452s) [noise=true , address=145.79.227.30:4785 , piece_id=EVFF6J2FLJNH6YGLWDGJCPCSJTLWM473ZJVLJDTT4OF4XYGGQPUQ, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:47.104195012 (504h3m46.975278047s) [noise=true , address=145.79.227.40:4059 , piece_id=WOW4SDDHF2FV3472XE6LLFWJOL2LCVGWTYOJ6ZECDY5OQK6LXGTA, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:47.130868082 (504h3m47.001951117s) [noise=true , address=145.79.227.34:4927 , piece_id=MHQ5HOSRWVH4MN5PQC6ABC2XZWZ3BMAJ6BOW4PVEZ352X3IJR3YQ, seg_pos={0 0}, upload=442332]: Done uploading piece. err: <nil> Dec 26 15:32:47.130886237 (504h3m47.001969272s) [seg_pos={0 0}, upload=442332]: Segment reached optimal threshold of 14 pieces. Dec 26 15:32:47.131077551 (504h3m47.002160586s) [noise=true , address=77.90.31.17:4792 , piece_id=SEZ64NSDSE7PFCGOWHVZSPJHYOXK4AGUJTTO6YBYXMYPSZH2PBOQ, seg_pos={0 0}, upload=442332]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SPFgYZnbwjEupjaghfs61qwFM678KNPwpDmRNqV1j2opNgZCN): protocol: expected piece hash; context canceled; EOF Dec 26 15:32:47.131083923 (504h3m47.002166958s) [noise=true , address=77.90.31.17:4792 , piece_id=SEZ64NSDSE7PFCGOWHVZSPJHYOXK4AGUJTTO6YBYXMYPSZH2PBOQ, seg_pos={0 0}, upload=442332]: Long tail cancellation due to optimal threshold Dec 26 15:32:47.131194734 (504h3m47.002277779s) [noise=true , address=45.91.116.10:4097 , piece_id=HOWSTA5KFZ7NGRXZPDDSZD2BFFFDZTJUTJL3GEWZ5GHCKB2AUTFQ, seg_pos={0 0}, upload=442332]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HcwSEGPEpd59dUukENRoUykRyqmVLiGcKEBbJ72qr9b5tpoEY): protocol: expected piece hash; context canceled; EOF Dec 26 15:32:47.131200846 (504h3m47.002283881s) [noise=true , address=45.91.116.10:4097 , piece_id=HOWSTA5KFZ7NGRXZPDDSZD2BFFFDZTJUTJL3GEWZ5GHCKB2AUTFQ, seg_pos={0 0}, upload=442332]: Long tail cancellation due to optimal threshold Dec 26 15:32:47.131313601 (504h3m47.002396646s) [noise=true , address=64.49.13.139:4473 , piece_id=KGXLN6OSUDXEIQ6CQG3UFQS3OKUBOLZX2UYANBWJODB2UILUEHIA, seg_pos={0 0}, upload=442332]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vE9uZrGbtGAPxJtzwoowofHwtp3d5MeeJw7qzAtSCdzcrxzKf): protocol: expected piece hash; context canceled; EOF Dec 26 15:32:47.131335142 (504h3m47.002418187s) [noise=true , address=64.49.13.139:4473 , piece_id=KGXLN6OSUDXEIQ6CQG3UFQS3OKUBOLZX2UYANBWJODB2UILUEHIA, seg_pos={0 0}, upload=442332]: Long tail cancellation due to optimal threshold Dec 26 15:32:47.131370139 (504h3m47.002453184s) [noise=true , address=64.49.13.169:4706 , piece_id=PY4667A2DCONGTD2CGFZ6I3DZON3JGBPEDAFRECAWGPND7S2P45A, seg_pos={0 0}, upload=442332]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:42364->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:42364->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42364->64.49.13.169:4706: use of closed network connection Dec 26 15:32:47.131392120 (504h3m47.002475165s) [noise=true , address=64.49.13.169:4706 , piece_id=PY4667A2DCONGTD2CGFZ6I3DZON3JGBPEDAFRECAWGPND7S2P45A, seg_pos={0 0}, upload=442332]: Long tail cancellation due to optimal threshold Dec 26 15:32:47.131426355 (504h3m47.00250939s) [seg_pos={0 0}, upload=442332]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:32:47.131518822 (504h3m47.002601867s) [upload=442332]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:32:47.131526266 (504h3m47.002609301s) [upload=442332]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:32:47.444749867 (504h3m47.315832902s) [upload=442332]: Done uploading segments... Dec 26 15:32:47.444776098 (504h3m47.315859143s) [upload=442332]: Upload finished. err: <nil> Duration: 1m0.001750624s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [..XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X.....X.X] duration: 5.001040845s trace: https://jaeger.dev.storj.io/trace/71972eeda6034f51 FAIL (2025-12-26 15:33:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001063358s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [......XX..X.X.....X.....X.X............XXX....XX...X......X.] duration: 5.841773095s trace: https://jaeger.dev.storj.io/trace/db5d56fbf5ab72f FAIL (2025-12-26 15:33:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 327]: Starting upload Dec 26 15:32:27.150869037 (504h3m27.021952082s) [upload=442327]: Done starting upload Dec 26 15:32:27.150966753 (504h3m27.022049798s) [upload=442327]: Uploading segments... Dec 26 15:32:27.151111518 (504h3m27.022194563s) [upload=442327]: Got next segment. Inline: false Dec 26 15:32:27.151160903 (504h3m27.022243938s) [upload=442327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:32:27.151170230 (504h3m27.022253265s) [upload=442327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:32:27.396820902 (504h3m27.267903947s) [seg_pos={0 0}, upload=442327]: Begin upload segment... Dec 26 15:32:27.396878382 (504h3m27.267961427s) [seg_pos={0 0}, upload=442327]: Done begin upload segment. Dec 26 15:32:27.396911815 (504h3m27.267994851s) [noise=true , address=45.91.116.7:4010 , piece_id=BIKQMUUUFMHSCKHXV3J2U4JRQKN67YM4VCOKYGRTP2GJKF57OUCA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.397064496 (504h3m27.268147531s) [noise=true , address=145.79.227.14:4837 , piece_id=7YCJGNJIGCM73ELVSK4VXO4HMFLHATCDIRZCCONCHZHXZIAZKD5Q, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.397617349 (504h3m27.268700414s) [noise=true , address=145.79.227.37:4567 , piece_id=4I2XA5ZFZ7MLVQN2JDLOWTKGXLSACCFJWPY7CTEHSXK424H3F6OA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398128021 (504h3m27.269211056s) [noise=true , address=145.79.227.15:4873 , piece_id=P6PRZN3BU7ZQITRIP2CWXP3MXAT35YUQEQHBA6MZCH3CGSYW6PXQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398146155 (504h3m27.26922919s) [noise=true , address=145.79.227.41:4085 , piece_id=5F3L4TMU2ZUHQHVNYZNEGNZEAJEHX6NWEK2OUXP6CICSGAEVNTMA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398458009 (504h3m27.269541044s) [noise=true , address=145.79.227.12:4767 , piece_id=TWYZFE7ANYIMNH3HF4E6OQDADJRM2OVJOKRQS6ETI36VKRRUM57A, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398467878 (504h3m27.269550923s) [noise=true , address=145.79.227.42:4123 , piece_id=PFMH6NY3GNEY5ZNMWMHAOXOQLFTZGLWNJPQNDUGHHOVTT3OI4PBQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398732943 (504h3m27.269815978s) [noise=true , address=145.79.227.50:4373 , piece_id=KB6YTMU5L7Z45LPRIZLEKBXTGBJM2AP74T3MYATZ6OY62J2IRAXA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398758772 (504h3m27.269841807s) [noise=true , address=45.91.116.14:4192 , piece_id=C22A5VXQ5CGXRP6SUFAJJTKYJQU6KRDSWTABIQGTDQ2JBG33GNPA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399028857 (504h3m27.270111902s) [noise=true , address=77.90.31.11:4555 , piece_id=SKCTG3FCS24RPX7RHIOHRKE77IA7WVLLLHAUJPWAH2CRX4227GEQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399043615 (504h3m27.27012666s) [noise=true , address=145.79.227.45:4231 , piece_id=YKOCOIXMMBVHHNICPFVIRPMEQR35M5TDALKKV4DUQSYQBEHI4AUQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399316584 (504h3m27.27039963s) [noise=true , address=145.79.227.52:4445 , piece_id=3NYGEGKMMV4M5WOK65YK7PGVONTEYD3MP7WP5TR2RRIZBREESTSA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399323958 (504h3m27.270407003s) [noise=true , address=45.91.116.11:4117 , piece_id=ZMEIAXIZPJ4OUBYYOTARZUYYZVREXS6S6I5LPKZEEEF23A6CQBWQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399593432 (504h3m27.270676467s) [noise=true , address=145.79.227.7:4294 , piece_id=G4FDHQR4EY22BSGCFMUW4PC2DVWY2H3EPPGPISZSF5TJFQBBT2EA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399869507 (504h3m27.270952543s) [noise=true , address=145.79.227.24:4581 , piece_id=6RP6INDHP4HNZBINR4JTSUU2WU2RV6NIOPXYIB564HCJCPHLB7WQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.400755334 (504h3m27.271838369s) [noise=true , address=77.90.31.14:4664 , piece_id=AQMWBOSHLHIP3R54PH2GULF5NPUR5VNB75SCWNMFQCGLSBZ7DHQQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:29.102300563 (504h3m28.973383608s) [upload=442327]: Next returned nil segment Dec 26 15:32:29.102325200 (504h3m28.973408235s) [upload=442327]: Waiting for error group managing segments... Dec 26 15:32:29.388594837 (504h3m29.259677872s) [noise=true , address=145.79.227.7:4294 , piece_id=G4FDHQR4EY22BSGCFMUW4PC2DVWY2H3EPPGPISZSF5TJFQBBT2EA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.389572970 (504h3m29.260656015s) [noise=true , address=145.79.227.50:4373 , piece_id=KB6YTMU5L7Z45LPRIZLEKBXTGBJM2AP74T3MYATZ6OY62J2IRAXA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.400604706 (504h3m29.271687741s) [noise=true , address=145.79.227.15:4873 , piece_id=P6PRZN3BU7ZQITRIP2CWXP3MXAT35YUQEQHBA6MZCH3CGSYW6PXQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.403736069 (504h3m29.274819114s) [noise=true , address=145.79.227.14:4837 , piece_id=7YCJGNJIGCM73ELVSK4VXO4HMFLHATCDIRZCCONCHZHXZIAZKD5Q, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.404653175 (504h3m29.27573621s) [noise=true , address=145.79.227.24:4581 , piece_id=6RP6INDHP4HNZBINR4JTSUU2WU2RV6NIOPXYIB564HCJCPHLB7WQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.407490368 (504h3m29.278573403s) [noise=true , address=145.79.227.42:4123 , piece_id=PFMH6NY3GNEY5ZNMWMHAOXOQLFTZGLWNJPQNDUGHHOVTT3OI4PBQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.408188768 (504h3m29.279271803s) [noise=true , address=145.79.227.41:4085 , piece_id=5F3L4TMU2ZUHQHVNYZNEGNZEAJEHX6NWEK2OUXP6CICSGAEVNTMA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.410656246 (504h3m29.281739281s) [noise=true , address=145.79.227.52:4445 , piece_id=3NYGEGKMMV4M5WOK65YK7PGVONTEYD3MP7WP5TR2RRIZBREESTSA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.410946158 (504h3m29.282029193s) [noise=true , address=145.79.227.37:4567 , piece_id=4I2XA5ZFZ7MLVQN2JDLOWTKGXLSACCFJWPY7CTEHSXK424H3F6OA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.413745288 (504h3m29.284828334s) [noise=true , address=145.79.227.45:4231 , piece_id=YKOCOIXMMBVHHNICPFVIRPMEQR35M5TDALKKV4DUQSYQBEHI4AUQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.425009076 (504h3m29.296092111s) [noise=true , address=145.79.227.12:4767 , piece_id=TWYZFE7ANYIMNH3HF4E6OQDADJRM2OVJOKRQS6ETI36VKRRUM57A, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.670341572 (504h3m29.541424607s) [noise=true , address=45.91.116.11:4117 , piece_id=ZMEIAXIZPJ4OUBYYOTARZUYYZVREXS6S6I5LPKZEEEF23A6CQBWQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.713495595 (504h3m29.58457863s) [noise=true , address=45.91.116.7:4010 , piece_id=BIKQMUUUFMHSCKHXV3J2U4JRQKN67YM4VCOKYGRTP2GJKF57OUCA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.713513468 (504h3m29.584596514s) [seg_pos={0 0}, upload=442327]: Segment reached optimal threshold of 13 pieces. Dec 26 15:32:29.713767834 (504h3m29.584850879s) [noise=true , address=45.91.116.14:4192 , piece_id=C22A5VXQ5CGXRP6SUFAJJTKYJQU6KRDSWTABIQGTDQ2JBG33GNPA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:55362->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:55362->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55362->45.91.116.14:4192: use of closed network connection Dec 26 15:32:29.713777161 (504h3m29.584860206s) [noise=true , address=45.91.116.14:4192 , piece_id=C22A5VXQ5CGXRP6SUFAJJTKYJQU6KRDSWTABIQGTDQ2JBG33GNPA, seg_pos={0 0}, upload=442327]: Long tail cancellation due to optimal threshold Dec 26 15:32:29.714024021 (504h3m29.585107066s) [noise=true , address=77.90.31.11:4555 , piece_id=SKCTG3FCS24RPX7RHIOHRKE77IA7WVLLLHAUJPWAH2CRX4227GEQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: read tcp 188.166.225.170:38774->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:32:29.714034621 (504h3m29.585117666s) [noise=true , address=77.90.31.11:4555 , piece_id=SKCTG3FCS24RPX7RHIOHRKE77IA7WVLLLHAUJPWAH2CRX4227GEQ, seg_pos={0 0}, upload=442327]: Long tail cancellation due to optimal threshold Dec 26 15:32:29.714088314 (504h3m29.585171359s) [noise=true , address=77.90.31.14:4664 , piece_id=AQMWBOSHLHIP3R54PH2GULF5NPUR5VNB75SCWNMFQCGLSBZ7DHQQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 26 15:32:29.714097962 (504h3m29.585181007s) [noise=true , address=77.90.31.14:4664 , piece_id=AQMWBOSHLHIP3R54PH2GULF5NPUR5VNB75SCWNMFQCGLSBZ7DHQQ, seg_pos={0 0}, upload=442327]: Long tail cancellation due to optimal threshold Dec 26 15:32:29.714116087 (504h3m29.585199122s) [seg_pos={0 0}, upload=442327]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:32:29.714198333 (504h3m29.585281408s) [upload=442327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:32:29.714207089 (504h3m29.585290124s) [upload=442327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:32:30.011069438 (504h3m29.882152473s) [upload=442327]: Done uploading segments... Dec 26 15:32:30.011090958 (504h3m29.882174003s) [upload=442327]: Upload finished. err: <nil> Duration: 5.841800847s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......XX..X.X.....X.....X.X............XXX....XX...X......X] duration: 30.003293848s trace: https://jaeger.dev.storj.io/trace/db5d56fbf5ab72f FAIL (2025-12-26 15:32:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 327]: Starting upload Dec 26 15:32:27.150869037 (504h3m27.021952082s) [upload=442327]: Done starting upload Dec 26 15:32:27.150966753 (504h3m27.022049798s) [upload=442327]: Uploading segments... Dec 26 15:32:27.151111518 (504h3m27.022194563s) [upload=442327]: Got next segment. Inline: false Dec 26 15:32:27.151160903 (504h3m27.022243938s) [upload=442327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:32:27.151170230 (504h3m27.022253265s) [upload=442327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:32:27.396820902 (504h3m27.267903947s) [seg_pos={0 0}, upload=442327]: Begin upload segment... Dec 26 15:32:27.396878382 (504h3m27.267961427s) [seg_pos={0 0}, upload=442327]: Done begin upload segment. Dec 26 15:32:27.396911815 (504h3m27.267994851s) [noise=true , address=45.91.116.7:4010 , piece_id=BIKQMUUUFMHSCKHXV3J2U4JRQKN67YM4VCOKYGRTP2GJKF57OUCA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.397064496 (504h3m27.268147531s) [noise=true , address=145.79.227.14:4837 , piece_id=7YCJGNJIGCM73ELVSK4VXO4HMFLHATCDIRZCCONCHZHXZIAZKD5Q, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.397617349 (504h3m27.268700414s) [noise=true , address=145.79.227.37:4567 , piece_id=4I2XA5ZFZ7MLVQN2JDLOWTKGXLSACCFJWPY7CTEHSXK424H3F6OA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398128021 (504h3m27.269211056s) [noise=true , address=145.79.227.15:4873 , piece_id=P6PRZN3BU7ZQITRIP2CWXP3MXAT35YUQEQHBA6MZCH3CGSYW6PXQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398146155 (504h3m27.26922919s) [noise=true , address=145.79.227.41:4085 , piece_id=5F3L4TMU2ZUHQHVNYZNEGNZEAJEHX6NWEK2OUXP6CICSGAEVNTMA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398458009 (504h3m27.269541044s) [noise=true , address=145.79.227.12:4767 , piece_id=TWYZFE7ANYIMNH3HF4E6OQDADJRM2OVJOKRQS6ETI36VKRRUM57A, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398467878 (504h3m27.269550923s) [noise=true , address=145.79.227.42:4123 , piece_id=PFMH6NY3GNEY5ZNMWMHAOXOQLFTZGLWNJPQNDUGHHOVTT3OI4PBQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398732943 (504h3m27.269815978s) [noise=true , address=145.79.227.50:4373 , piece_id=KB6YTMU5L7Z45LPRIZLEKBXTGBJM2AP74T3MYATZ6OY62J2IRAXA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.398758772 (504h3m27.269841807s) [noise=true , address=45.91.116.14:4192 , piece_id=C22A5VXQ5CGXRP6SUFAJJTKYJQU6KRDSWTABIQGTDQ2JBG33GNPA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399028857 (504h3m27.270111902s) [noise=true , address=77.90.31.11:4555 , piece_id=SKCTG3FCS24RPX7RHIOHRKE77IA7WVLLLHAUJPWAH2CRX4227GEQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399043615 (504h3m27.27012666s) [noise=true , address=145.79.227.45:4231 , piece_id=YKOCOIXMMBVHHNICPFVIRPMEQR35M5TDALKKV4DUQSYQBEHI4AUQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399316584 (504h3m27.27039963s) [noise=true , address=145.79.227.52:4445 , piece_id=3NYGEGKMMV4M5WOK65YK7PGVONTEYD3MP7WP5TR2RRIZBREESTSA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399323958 (504h3m27.270407003s) [noise=true , address=45.91.116.11:4117 , piece_id=ZMEIAXIZPJ4OUBYYOTARZUYYZVREXS6S6I5LPKZEEEF23A6CQBWQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399593432 (504h3m27.270676467s) [noise=true , address=145.79.227.7:4294 , piece_id=G4FDHQR4EY22BSGCFMUW4PC2DVWY2H3EPPGPISZSF5TJFQBBT2EA, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.399869507 (504h3m27.270952543s) [noise=true , address=145.79.227.24:4581 , piece_id=6RP6INDHP4HNZBINR4JTSUU2WU2RV6NIOPXYIB564HCJCPHLB7WQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:27.400755334 (504h3m27.271838369s) [noise=true , address=77.90.31.14:4664 , piece_id=AQMWBOSHLHIP3R54PH2GULF5NPUR5VNB75SCWNMFQCGLSBZ7DHQQ, seg_pos={0 0}, upload=442327]: Uploading piece... Dec 26 15:32:29.102300563 (504h3m28.973383608s) [upload=442327]: Next returned nil segment Dec 26 15:32:29.102325200 (504h3m28.973408235s) [upload=442327]: Waiting for error group managing segments... Dec 26 15:32:29.388594837 (504h3m29.259677872s) [noise=true , address=145.79.227.7:4294 , piece_id=G4FDHQR4EY22BSGCFMUW4PC2DVWY2H3EPPGPISZSF5TJFQBBT2EA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.389572970 (504h3m29.260656015s) [noise=true , address=145.79.227.50:4373 , piece_id=KB6YTMU5L7Z45LPRIZLEKBXTGBJM2AP74T3MYATZ6OY62J2IRAXA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.400604706 (504h3m29.271687741s) [noise=true , address=145.79.227.15:4873 , piece_id=P6PRZN3BU7ZQITRIP2CWXP3MXAT35YUQEQHBA6MZCH3CGSYW6PXQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.403736069 (504h3m29.274819114s) [noise=true , address=145.79.227.14:4837 , piece_id=7YCJGNJIGCM73ELVSK4VXO4HMFLHATCDIRZCCONCHZHXZIAZKD5Q, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.404653175 (504h3m29.27573621s) [noise=true , address=145.79.227.24:4581 , piece_id=6RP6INDHP4HNZBINR4JTSUU2WU2RV6NIOPXYIB564HCJCPHLB7WQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.407490368 (504h3m29.278573403s) [noise=true , address=145.79.227.42:4123 , piece_id=PFMH6NY3GNEY5ZNMWMHAOXOQLFTZGLWNJPQNDUGHHOVTT3OI4PBQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.408188768 (504h3m29.279271803s) [noise=true , address=145.79.227.41:4085 , piece_id=5F3L4TMU2ZUHQHVNYZNEGNZEAJEHX6NWEK2OUXP6CICSGAEVNTMA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.410656246 (504h3m29.281739281s) [noise=true , address=145.79.227.52:4445 , piece_id=3NYGEGKMMV4M5WOK65YK7PGVONTEYD3MP7WP5TR2RRIZBREESTSA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.410946158 (504h3m29.282029193s) [noise=true , address=145.79.227.37:4567 , piece_id=4I2XA5ZFZ7MLVQN2JDLOWTKGXLSACCFJWPY7CTEHSXK424H3F6OA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.413745288 (504h3m29.284828334s) [noise=true , address=145.79.227.45:4231 , piece_id=YKOCOIXMMBVHHNICPFVIRPMEQR35M5TDALKKV4DUQSYQBEHI4AUQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.425009076 (504h3m29.296092111s) [noise=true , address=145.79.227.12:4767 , piece_id=TWYZFE7ANYIMNH3HF4E6OQDADJRM2OVJOKRQS6ETI36VKRRUM57A, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.670341572 (504h3m29.541424607s) [noise=true , address=45.91.116.11:4117 , piece_id=ZMEIAXIZPJ4OUBYYOTARZUYYZVREXS6S6I5LPKZEEEF23A6CQBWQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.713495595 (504h3m29.58457863s) [noise=true , address=45.91.116.7:4010 , piece_id=BIKQMUUUFMHSCKHXV3J2U4JRQKN67YM4VCOKYGRTP2GJKF57OUCA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: <nil> Dec 26 15:32:29.713513468 (504h3m29.584596514s) [seg_pos={0 0}, upload=442327]: Segment reached optimal threshold of 13 pieces. Dec 26 15:32:29.713767834 (504h3m29.584850879s) [noise=true , address=45.91.116.14:4192 , piece_id=C22A5VXQ5CGXRP6SUFAJJTKYJQU6KRDSWTABIQGTDQ2JBG33GNPA, seg_pos={0 0}, upload=442327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:55362->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:55362->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55362->45.91.116.14:4192: use of closed network connection Dec 26 15:32:29.713777161 (504h3m29.584860206s) [noise=true , address=45.91.116.14:4192 , piece_id=C22A5VXQ5CGXRP6SUFAJJTKYJQU6KRDSWTABIQGTDQ2JBG33GNPA, seg_pos={0 0}, upload=442327]: Long tail cancellation due to optimal threshold Dec 26 15:32:29.714024021 (504h3m29.585107066s) [noise=true , address=77.90.31.11:4555 , piece_id=SKCTG3FCS24RPX7RHIOHRKE77IA7WVLLLHAUJPWAH2CRX4227GEQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: read tcp 188.166.225.170:38774->77.90.31.11:4555: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:32:29.714034621 (504h3m29.585117666s) [noise=true , address=77.90.31.11:4555 , piece_id=SKCTG3FCS24RPX7RHIOHRKE77IA7WVLLLHAUJPWAH2CRX4227GEQ, seg_pos={0 0}, upload=442327]: Long tail cancellation due to optimal threshold Dec 26 15:32:29.714088314 (504h3m29.585171359s) [noise=true , address=77.90.31.14:4664 , piece_id=AQMWBOSHLHIP3R54PH2GULF5NPUR5VNB75SCWNMFQCGLSBZ7DHQQ, seg_pos={0 0}, upload=442327]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 26 15:32:29.714097962 (504h3m29.585181007s) [noise=true , address=77.90.31.14:4664 , piece_id=AQMWBOSHLHIP3R54PH2GULF5NPUR5VNB75SCWNMFQCGLSBZ7DHQQ, seg_pos={0 0}, upload=442327]: Long tail cancellation due to optimal threshold Dec 26 15:32:29.714116087 (504h3m29.585199122s) [seg_pos={0 0}, upload=442327]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:32:29.714198333 (504h3m29.585281408s) [upload=442327]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:32:29.714207089 (504h3m29.585290124s) [upload=442327]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:32:30.011069438 (504h3m29.882152473s) [upload=442327]: Done uploading segments... Dec 26 15:32:30.011090958 (504h3m29.882174003s) [upload=442327]: Upload finished. err: <nil> Duration: 30.00331642s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [X..XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X.....X.] duration: 5.001126608s trace: https://jaeger.dev.storj.io/trace/421007474598b989 FAIL (2025-12-26 15:32:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001156124s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [..XX..X.X..X..X.X...XX.XXXX.X..................X..........X.] duration: 42.904779997s trace: https://jaeger.dev.storj.io/trace/32a61e9d8fd035d7 FAIL (2025-12-26 15:32:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, Piece ID: FJYRNOONQMIIDM6VBAVZOFF6E77TRETBZXTVMUVQATRV44U5VYIA) context deadline exceeded; piecestore: (Node ID: 1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, Piece ID: 47HIT7FJPRNLC7VZ2VVJ5TVE2LNTEDA5RLSBB47VEJGYVGIU5BGA) context deadline exceeded; piecestore: (Node ID: 12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, Piece ID: 7SA4CA6QP2LJECF5KOB2AABX7SQRLF23FBEZJP5RSISXQ4KYAK4Q) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: ODAIX77EGMEEVAQKESXUBW7IT7TDI4FZ4UV6N64FOHDV7J3HVXDA) context deadline exceeded ======LOGS====== DVPA7QQ7BFGH2N3X5IIH5TCT2G57LSBT6WEUV42ZN4XJQA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.771300987 (504h1m34.642384022s) [noise=true , address=77.90.31.18:4827 , piece_id=ODAIX77EGMEEVAQKESXUBW7IT7TDI4FZ4UV6N64FOHDV7J3HVXDA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.771559299 (504h1m34.642642334s) [noise=true , address=145.79.227.40:4067 , piece_id=5KE2JMY5KC6TCONKFUIBNVS4AUDNKQMG65E3UTJYTW5D7AFYNUXA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.771874770 (504h1m34.642957815s) [noise=true , address=45.91.116.12:4246 , piece_id=FJYRNOONQMIIDM6VBAVZOFF6E77TRETBZXTVMUVQATRV44U5VYIA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.772132260 (504h1m34.643215306s) [noise=true , address=193.149.164.143:4702 , piece_id=3F5PVK3A7EYO4V7JGFSK2NL22Q4E53YR77LE5DIRYZ4ISNUTSVZA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.772457289 (504h1m34.643540324s) [noise=true , address=77.90.31.17:4803 , piece_id=NDIADSENGEJXVRTVNDJUMHC4W3NIGUUJNPJWDOY5LGEHKSWFDCGQ, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.772731441 (504h1m34.643814486s) [noise=true , address=77.90.31.13:4660 , piece_id=47HIT7FJPRNLC7VZ2VVJ5TVE2LNTEDA5RLSBB47VEJGYVGIU5BGA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.772992498 (504h1m34.644075543s) [noise=true , address=145.79.227.8:4333 , piece_id=L36MOOQAQTXZV7VWPRUCDCBNL3PJWBNKHPCCUMXECAW2KBT7LWDQ, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.773387410 (504h1m34.644470445s) [noise=true , address=77.90.31.10:4535 , piece_id=7SA4CA6QP2LJECF5KOB2AABX7SQRLF23FBEZJP5RSISXQ4KYAK4Q, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.773541223 (504h1m34.644624258s) [noise=true , address=64.49.13.169:4710 , piece_id=W6AZE2H64A6747ZLYAFYGU6MISWQUZIYUJJFNXOMJBW3C4RQR7EQ, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.773891450 (504h1m34.644974485s) [noise=true , address=145.79.227.44:4212 , piece_id=KQW2MXPIK6D4PICHI5SVQHUJQNMTNU2F46F6OOD5JLRVENDQ4IHA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.774208523 (504h1m34.645291568s) [noise=true , address=77.90.31.16:4737 , piece_id=HJ3S2T35ZWDUNNDDPNX7VFDNYVVUVAM47L5O53XG7J73MFAIQVIA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.774519075 (504h1m34.64560211s) [noise=true , address=45.91.116.10:4097 , piece_id=AXN5TAQROOD5QKXPZ4GX43RUPQ7S3NOIIQG7HO3JD632MLSSWXFA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.774887877 (504h1m34.645970923s) [noise=true , address=77.90.31.15:4729 , piece_id=VWUILBBBP2TA5YXYLYK4QWJAYBQGD3QSMC7JPNRZW4UCF77Q63IA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.777008906 (504h1m34.648091941s) [noise=true , address=77.90.31.11:4560 , piece_id=XEHGTZEINWBJDVMVEO64UFEWFH2YA2IMO5W5J4B7WUGDLXCJMGVQ, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.777437983 (504h1m34.648521018s) [noise=true , address=45.91.116.6:4985 , piece_id=CMQ25ELNYVRW35JY6TM3SV3I4CIPNO5JVE546COPMXSFI52KV4QA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:36.066992931 (504h1m35.938075976s) [upload=442304]: Next returned nil segment Dec 26 15:30:36.067014472 (504h1m35.938097507s) [upload=442304]: Waiting for error group managing segments... Dec 26 15:30:36.394321622 (504h1m36.265404657s) [noise=true , address=193.149.164.143:4702 , piece_id=3F5PVK3A7EYO4V7JGFSK2NL22Q4E53YR77LE5DIRYZ4ISNUTSVZA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:36.715899321 (504h1m36.586982366s) [noise=true , address=145.79.227.8:4333 , piece_id=L36MOOQAQTXZV7VWPRUCDCBNL3PJWBNKHPCCUMXECAW2KBT7LWDQ, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:36.899071434 (504h1m36.770154469s) [noise=true , address=77.90.31.16:4737 , piece_id=HJ3S2T35ZWDUNNDDPNX7VFDNYVVUVAM47L5O53XG7J73MFAIQVIA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:36.911221970 (504h1m36.782305005s) [noise=true , address=45.91.116.12:4246 , piece_id=FJYRNOONQMIIDM6VBAVZOFF6E77TRETBZXTVMUVQATRV44U5VYIA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:36.996173923 (504h1m36.867256968s) [noise=true , address=45.91.116.10:4097 , piece_id=AXN5TAQROOD5QKXPZ4GX43RUPQ7S3NOIIQG7HO3JD632MLSSWXFA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:37.025791320 (504h1m36.896874355s) [noise=true , address=77.90.31.12:4595 , piece_id=XP3XHIXAA7H74PW4LAQADU4JB3LLG57HNQIXBYOLO5DVYL2MCNUA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:37.077963435 (504h1m36.94904648s) [noise=true , address=77.90.31.13:4660 , piece_id=47HIT7FJPRNLC7VZ2VVJ5TVE2LNTEDA5RLSBB47VEJGYVGIU5BGA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:37.552985670 (504h1m37.424068706s) [noise=true , address=145.79.227.44:4212 , piece_id=KQW2MXPIK6D4PICHI5SVQHUJQNMTNU2F46F6OOD5JLRVENDQ4IHA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:37.577700872 (504h1m37.448783908s) [noise=true , address=77.90.31.10:4535 , piece_id=7SA4CA6QP2LJECF5KOB2AABX7SQRLF23FBEZJP5RSISXQ4KYAK4Q, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:37.605536707 (504h1m37.476619742s) [noise=true , address=77.90.31.18:4827 , piece_id=ODAIX77EGMEEVAQKESXUBW7IT7TDI4FZ4UV6N64FOHDV7J3HVXDA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:37.628071597 (504h1m37.499154642s) [noise=true , address=145.79.227.30:4788 , piece_id=ZMDWHDDVPA7QQ7BFGH2N3X5IIH5TCT2G57LSBT6WEUV42ZN4XJQA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:37.975673933 (504h1m37.846756968s) [noise=true , address=145.79.227.40:4067 , piece_id=5KE2JMY5KC6TCONKFUIBNVS4AUDNKQMG65E3UTJYTW5D7AFYNUXA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:37.990248185 (504h1m37.86133123s) [noise=true , address=77.90.31.15:4729 , piece_id=VWUILBBBP2TA5YXYLYK4QWJAYBQGD3QSMC7JPNRZW4UCF77Q63IA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:38.072929144 (504h1m37.944012189s) [noise=true , address=64.49.13.137:4495 , piece_id=AG4SUNALTTG5AZ7Z45VZ7VWZBFBMIQSLYIYCHDSYOCANV3TQGRGQ, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:38.072975903 (504h1m37.944058948s) [seg_pos={0 0}, upload=442304]: Segment reached optimal threshold of 14 pieces. Dec 26 15:30:38.073453272 (504h1m37.944536317s) [noise=true , address=45.91.116.6:4985 , piece_id=CMQ25ELNYVRW35JY6TM3SV3I4CIPNO5JVE546COPMXSFI52KV4QA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: read tcp 188.166.225.170:54480->45.91.116.6:4985: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:30:38.073462369 (504h1m37.944545404s) [noise=true , address=45.91.116.6:4985 , piece_id=CMQ25ELNYVRW35JY6TM3SV3I4CIPNO5JVE546COPMXSFI52KV4QA, seg_pos={0 0}, upload=442304]: Long tail cancellation due to optimal threshold Dec 26 15:30:38.073639436 (504h1m37.944722471s) [noise=true , address=77.90.31.11:4560 , piece_id=XEHGTZEINWBJDVMVEO64UFEWFH2YA2IMO5W5J4B7WUGDLXCJMGVQ, seg_pos={0 0}, upload=442304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH): protocol: expected piece hash; context canceled; EOF Dec 26 15:30:38.073645929 (504h1m37.944728974s) [noise=true , address=77.90.31.11:4560 , piece_id=XEHGTZEINWBJDVMVEO64UFEWFH2YA2IMO5W5J4B7WUGDLXCJMGVQ, seg_pos={0 0}, upload=442304]: Long tail cancellation due to optimal threshold Dec 26 15:30:38.073705773 (504h1m37.944788828s) [noise=true , address=64.49.13.169:4710 , piece_id=W6AZE2H64A6747ZLYAFYGU6MISWQUZIYUJJFNXOMJBW3C4RQR7EQ, seg_pos={0 0}, upload=442304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:36438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:36438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36438->64.49.13.169:4710: use of closed network connection Dec 26 15:30:38.073759826 (504h1m37.944842871s) [noise=true , address=64.49.13.169:4710 , piece_id=W6AZE2H64A6747ZLYAFYGU6MISWQUZIYUJJFNXOMJBW3C4RQR7EQ, seg_pos={0 0}, upload=442304]: Long tail cancellation due to optimal threshold Dec 26 15:30:38.073885184 (504h1m37.944968239s) [noise=true , address=77.90.31.17:4803 , piece_id=NDIADSENGEJXVRTVNDJUMHC4W3NIGUUJNPJWDOY5LGEHKSWFDCGQ, seg_pos={0 0}, upload=442304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y): protocol: expected piece hash; context canceled; EOF Dec 26 15:30:38.073895354 (504h1m37.944978389s) [noise=true , address=77.90.31.17:4803 , piece_id=NDIADSENGEJXVRTVNDJUMHC4W3NIGUUJNPJWDOY5LGEHKSWFDCGQ, seg_pos={0 0}, upload=442304]: Long tail cancellation due to optimal threshold Dec 26 15:30:38.073923597 (504h1m37.945006632s) [seg_pos={0 0}, upload=442304]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:30:38.074025632 (504h1m37.945108677s) [upload=442304]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:30:38.074032806 (504h1m37.945115851s) [upload=442304]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:30:38.374718093 (504h1m38.245801128s) [upload=442304]: Done uploading segments... Dec 26 15:30:38.374741477 (504h1m38.245824512s) [upload=442304]: Upload finished. err: <nil> Duration: 42.904794907s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [.X..XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X.....X] duration: 5.001434283s trace: https://jaeger.dev.storj.io/trace/421007474598b989 FAIL (2025-12-26 15:32:16 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001454022s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [........XX..X.X.....X.....X.X............XXX....XX...X......] duration: 5.846876173s trace: https://jaeger.dev.storj.io/trace/295f48e94306f825 FAIL (2025-12-26 15:31:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ) context deadline exceeded ======LOGS====== Dec 26 15:26:13.096092554 (503h57m12.967175599s) [upload=442241]: Starting upload Dec 26 15:26:13.096201371 (503h57m12.967284426s) [upload=442241]: Done starting upload Dec 26 15:26:13.096285331 (503h57m12.967368366s) [upload=442241]: Uploading segments... Dec 26 15:26:13.096634736 (503h57m12.967717781s) [upload=442241]: Got next segment. Inline: false Dec 26 15:26:13.096720009 (503h57m12.967803054s) [upload=442241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:26:13.096731480 (503h57m12.967814555s) [upload=442241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:26:13.346386396 (503h57m13.217469431s) [seg_pos={0 0}, upload=442241]: Begin upload segment... Dec 26 15:26:13.346431882 (503h57m13.217514917s) [seg_pos={0 0}, upload=442241]: Done begin upload segment. Dec 26 15:26:13.346455677 (503h57m13.217538723s) [noise=true , address=145.79.227.4:4186 , piece_id=TQBUGQ6SPETEJTLK4AK63X5F5SSZIUU6QKQDVQFV4H25Z2W2V54Q, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.346518798 (503h57m13.217601843s) [noise=true , address=145.79.227.51:4410 , piece_id=CKMYU2WJBXZBNV36LNRODJDVRUWHKX4N7F6QONY3ORA7FOPJZVZQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347012878 (503h57m13.218095924s) [noise=true , address=45.91.116.14:4191 , piece_id=6SUIRACEDO357ODZ5ZYBX6HRXA6KH72M467CEZZTGAHMSXVKHAVQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347077742 (503h57m13.218160787s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347456193 (503h57m13.218539238s) [noise=true , address=45.91.116.5:4299 , piece_id=2GZLZBNFNIZ44QMW3PKNU65WWWUBX5ZSQFUKWGBCIAKDMIYCJS7A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347756103 (503h57m13.218839148s) [noise=true , address=45.91.116.10:4083 , piece_id=5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348078628 (503h57m13.219161703s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348395862 (503h57m13.219478907s) [noise=true , address=145.79.227.15:4876 , piece_id=YOOH7ACHJ2VL3ZJIZY3DL3ZUL42RCEKVFQDFZ22BXDEGLZWBPC5A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348733134 (503h57m13.219816179s) [noise=true , address=77.90.31.18:4808 , piece_id=WAUTGUPSCCXPIV4VENHXEOB57FWC7OQK3GN2UAHHQREX53RJXVAQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.349016083 (503h57m13.220099128s) [noise=true , address=45.91.116.6:4973 , piece_id=KF27ZZMNPB5PRCUMMWSCAVSYRLMNHSO7L2NHCRW3RGKZ7JOI4RFA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.351559577 (503h57m13.222642612s) [noise=true , address=145.79.227.39:4015 , piece_id=BMXFV5TLEC6WJBJFWJBITMAUWVTC3ZU56U7SIYLUTL3GXIEEMCNQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.351893382 (503h57m13.222976417s) [noise=true , address=145.79.227.48:4301 , piece_id=GVZGXRQUAF3TCT5GVQUSI6VT2XX6G5O6PUC4J26REL76VU5ZLALQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.352776163 (503h57m13.223859208s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.353302125 (503h57m13.22438517s) [noise=true , address=77.90.31.15:4700 , piece_id=DV5PGV2FOWNSONKYW5TVO2GEAHRKRFPH4VTQF2FISNKOQ7JFNEKQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.354596349 (503h57m13.225679394s) [noise=true , address=45.91.116.4:4940 , piece_id=CFSW3OH4JGHUTOWS2SR5ZQ6B7TZ5WQT7DGGD3K2ZUMLHEEEDWKHQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.355634927 (503h57m13.226717972s) [noise=true , address=45.91.116.11:4118 , piece_id=V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:15.062260982 (503h57m14.933344028s) [upload=442241]: Next returned nil segment Dec 26 15:26:15.062318943 (503h57m14.933401988s) [upload=442241]: Waiting for error group managing segments... Dec 26 15:26:15.344789265 (503h57m15.2158723s) [noise=true , address=145.79.227.4:4186 , piece_id=TQBUGQ6SPETEJTLK4AK63X5F5SSZIUU6QKQDVQFV4H25Z2W2V54Q, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.348322673 (503h57m15.219405708s) [noise=true , address=145.79.227.15:4876 , piece_id=YOOH7ACHJ2VL3ZJIZY3DL3ZUL42RCEKVFQDFZ22BXDEGLZWBPC5A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.360828747 (503h57m15.231911782s) [noise=true , address=145.79.227.51:4410 , piece_id=CKMYU2WJBXZBNV36LNRODJDVRUWHKX4N7F6QONY3ORA7FOPJZVZQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.370693872 (503h57m15.241776907s) [noise=true , address=145.79.227.39:4015 , piece_id=BMXFV5TLEC6WJBJFWJBITMAUWVTC3ZU56U7SIYLUTL3GXIEEMCNQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.384026740 (503h57m15.255109775s) [noise=true , address=145.79.227.48:4301 , piece_id=GVZGXRQUAF3TCT5GVQUSI6VT2XX6G5O6PUC4J26REL76VU5ZLALQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.604946208 (503h57m15.476029243s) [noise=true , address=45.91.116.5:4299 , piece_id=2GZLZBNFNIZ44QMW3PKNU65WWWUBX5ZSQFUKWGBCIAKDMIYCJS7A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.625370549 (503h57m15.496453584s) [noise=true , address=45.91.116.6:4973 , piece_id=KF27ZZMNPB5PRCUMMWSCAVSYRLMNHSO7L2NHCRW3RGKZ7JOI4RFA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.768372509 (503h57m15.639455544s) [noise=true , address=77.90.31.15:4700 , piece_id=DV5PGV2FOWNSONKYW5TVO2GEAHRKRFPH4VTQF2FISNKOQ7JFNEKQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.772904880 (503h57m15.643987925s) [noise=true , address=45.91.116.4:4940 , piece_id=CFSW3OH4JGHUTOWS2SR5ZQ6B7TZ5WQT7DGGD3K2ZUMLHEEEDWKHQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.776538659 (503h57m15.647621694s) [noise=true , address=45.91.116.11:4118 , piece_id=V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.825341898 (503h57m15.696424933s) [noise=true , address=45.91.116.14:4191 , piece_id=6SUIRACEDO357ODZ5ZYBX6HRXA6KH72M467CEZZTGAHMSXVKHAVQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.209263741 (503h57m16.080346786s) [noise=true , address=77.90.31.18:4808 , piece_id=WAUTGUPSCCXPIV4VENHXEOB57FWC7OQK3GN2UAHHQREX53RJXVAQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.481619854 (503h57m16.352702889s) [noise=true , address=45.91.116.10:4083 , piece_id=5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.481638810 (503h57m16.352721845s) [seg_pos={0 0}, upload=442241]: Segment reached optimal threshold of 13 pieces. Dec 26 15:26:16.481856614 (503h57m16.352939649s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.481863618 (503h57m16.352946653s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482081783 (503h57m16.353164819s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.482087865 (503h57m16.3531709s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482202613 (503h57m16.353285648s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.482208123 (503h57m16.353291169s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482224265 (503h57m16.3533073s) [seg_pos={0 0}, upload=442241]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:26:16.482285512 (503h57m16.353368547s) [upload=442241]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:26:16.482290240 (503h57m16.353373275s) [upload=442241]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:26:16.771425964 (503h57m16.642508999s) [upload=442241]: Done uploading segments... Dec 26 15:26:16.771446753 (503h57m16.642529798s) [upload=442241]: Upload finished. err: <nil> Duration: 5.846900829s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [X..XX..X.X..X..X.X...XX.XXXX.X..................X..........X] duration: 1m0.002060013s trace: https://jaeger.dev.storj.io/trace/32a61e9d8fd035d7 FAIL (2025-12-26 15:31:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ZzrVXFNGjn1h3v1otCVCKMtAxM2mzfUyCzBDVzqz12TRXHYp6, Piece ID: FJYRNOONQMIIDM6VBAVZOFF6E77TRETBZXTVMUVQATRV44U5VYIA) context deadline exceeded; piecestore: (Node ID: 1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, Piece ID: 47HIT7FJPRNLC7VZ2VVJ5TVE2LNTEDA5RLSBB47VEJGYVGIU5BGA) context deadline exceeded; piecestore: (Node ID: 12a3tsVj8R78geQJnTzdSroG6H6RiW3xv5YLyiBh1ctwSvAKeRz, Piece ID: 7SA4CA6QP2LJECF5KOB2AABX7SQRLF23FBEZJP5RSISXQ4KYAK4Q) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: ODAIX77EGMEEVAQKESXUBW7IT7TDI4FZ4UV6N64FOHDV7J3HVXDA) context deadline exceeded ======LOGS====== DVPA7QQ7BFGH2N3X5IIH5TCT2G57LSBT6WEUV42ZN4XJQA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.771300987 (504h1m34.642384022s) [noise=true , address=77.90.31.18:4827 , piece_id=ODAIX77EGMEEVAQKESXUBW7IT7TDI4FZ4UV6N64FOHDV7J3HVXDA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.771559299 (504h1m34.642642334s) [noise=true , address=145.79.227.40:4067 , piece_id=5KE2JMY5KC6TCONKFUIBNVS4AUDNKQMG65E3UTJYTW5D7AFYNUXA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.771874770 (504h1m34.642957815s) [noise=true , address=45.91.116.12:4246 , piece_id=FJYRNOONQMIIDM6VBAVZOFF6E77TRETBZXTVMUVQATRV44U5VYIA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.772132260 (504h1m34.643215306s) [noise=true , address=193.149.164.143:4702 , piece_id=3F5PVK3A7EYO4V7JGFSK2NL22Q4E53YR77LE5DIRYZ4ISNUTSVZA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.772457289 (504h1m34.643540324s) [noise=true , address=77.90.31.17:4803 , piece_id=NDIADSENGEJXVRTVNDJUMHC4W3NIGUUJNPJWDOY5LGEHKSWFDCGQ, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.772731441 (504h1m34.643814486s) [noise=true , address=77.90.31.13:4660 , piece_id=47HIT7FJPRNLC7VZ2VVJ5TVE2LNTEDA5RLSBB47VEJGYVGIU5BGA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.772992498 (504h1m34.644075543s) [noise=true , address=145.79.227.8:4333 , piece_id=L36MOOQAQTXZV7VWPRUCDCBNL3PJWBNKHPCCUMXECAW2KBT7LWDQ, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.773387410 (504h1m34.644470445s) [noise=true , address=77.90.31.10:4535 , piece_id=7SA4CA6QP2LJECF5KOB2AABX7SQRLF23FBEZJP5RSISXQ4KYAK4Q, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.773541223 (504h1m34.644624258s) [noise=true , address=64.49.13.169:4710 , piece_id=W6AZE2H64A6747ZLYAFYGU6MISWQUZIYUJJFNXOMJBW3C4RQR7EQ, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.773891450 (504h1m34.644974485s) [noise=true , address=145.79.227.44:4212 , piece_id=KQW2MXPIK6D4PICHI5SVQHUJQNMTNU2F46F6OOD5JLRVENDQ4IHA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.774208523 (504h1m34.645291568s) [noise=true , address=77.90.31.16:4737 , piece_id=HJ3S2T35ZWDUNNDDPNX7VFDNYVVUVAM47L5O53XG7J73MFAIQVIA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.774519075 (504h1m34.64560211s) [noise=true , address=45.91.116.10:4097 , piece_id=AXN5TAQROOD5QKXPZ4GX43RUPQ7S3NOIIQG7HO3JD632MLSSWXFA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.774887877 (504h1m34.645970923s) [noise=true , address=77.90.31.15:4729 , piece_id=VWUILBBBP2TA5YXYLYK4QWJAYBQGD3QSMC7JPNRZW4UCF77Q63IA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.777008906 (504h1m34.648091941s) [noise=true , address=77.90.31.11:4560 , piece_id=XEHGTZEINWBJDVMVEO64UFEWFH2YA2IMO5W5J4B7WUGDLXCJMGVQ, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:34.777437983 (504h1m34.648521018s) [noise=true , address=45.91.116.6:4985 , piece_id=CMQ25ELNYVRW35JY6TM3SV3I4CIPNO5JVE546COPMXSFI52KV4QA, seg_pos={0 0}, upload=442304]: Uploading piece... Dec 26 15:30:36.066992931 (504h1m35.938075976s) [upload=442304]: Next returned nil segment Dec 26 15:30:36.067014472 (504h1m35.938097507s) [upload=442304]: Waiting for error group managing segments... Dec 26 15:30:36.394321622 (504h1m36.265404657s) [noise=true , address=193.149.164.143:4702 , piece_id=3F5PVK3A7EYO4V7JGFSK2NL22Q4E53YR77LE5DIRYZ4ISNUTSVZA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:36.715899321 (504h1m36.586982366s) [noise=true , address=145.79.227.8:4333 , piece_id=L36MOOQAQTXZV7VWPRUCDCBNL3PJWBNKHPCCUMXECAW2KBT7LWDQ, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:36.899071434 (504h1m36.770154469s) [noise=true , address=77.90.31.16:4737 , piece_id=HJ3S2T35ZWDUNNDDPNX7VFDNYVVUVAM47L5O53XG7J73MFAIQVIA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:36.911221970 (504h1m36.782305005s) [noise=true , address=45.91.116.12:4246 , piece_id=FJYRNOONQMIIDM6VBAVZOFF6E77TRETBZXTVMUVQATRV44U5VYIA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:36.996173923 (504h1m36.867256968s) [noise=true , address=45.91.116.10:4097 , piece_id=AXN5TAQROOD5QKXPZ4GX43RUPQ7S3NOIIQG7HO3JD632MLSSWXFA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:37.025791320 (504h1m36.896874355s) [noise=true , address=77.90.31.12:4595 , piece_id=XP3XHIXAA7H74PW4LAQADU4JB3LLG57HNQIXBYOLO5DVYL2MCNUA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:37.077963435 (504h1m36.94904648s) [noise=true , address=77.90.31.13:4660 , piece_id=47HIT7FJPRNLC7VZ2VVJ5TVE2LNTEDA5RLSBB47VEJGYVGIU5BGA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:37.552985670 (504h1m37.424068706s) [noise=true , address=145.79.227.44:4212 , piece_id=KQW2MXPIK6D4PICHI5SVQHUJQNMTNU2F46F6OOD5JLRVENDQ4IHA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:37.577700872 (504h1m37.448783908s) [noise=true , address=77.90.31.10:4535 , piece_id=7SA4CA6QP2LJECF5KOB2AABX7SQRLF23FBEZJP5RSISXQ4KYAK4Q, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:37.605536707 (504h1m37.476619742s) [noise=true , address=77.90.31.18:4827 , piece_id=ODAIX77EGMEEVAQKESXUBW7IT7TDI4FZ4UV6N64FOHDV7J3HVXDA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:37.628071597 (504h1m37.499154642s) [noise=true , address=145.79.227.30:4788 , piece_id=ZMDWHDDVPA7QQ7BFGH2N3X5IIH5TCT2G57LSBT6WEUV42ZN4XJQA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:37.975673933 (504h1m37.846756968s) [noise=true , address=145.79.227.40:4067 , piece_id=5KE2JMY5KC6TCONKFUIBNVS4AUDNKQMG65E3UTJYTW5D7AFYNUXA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:37.990248185 (504h1m37.86133123s) [noise=true , address=77.90.31.15:4729 , piece_id=VWUILBBBP2TA5YXYLYK4QWJAYBQGD3QSMC7JPNRZW4UCF77Q63IA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:38.072929144 (504h1m37.944012189s) [noise=true , address=64.49.13.137:4495 , piece_id=AG4SUNALTTG5AZ7Z45VZ7VWZBFBMIQSLYIYCHDSYOCANV3TQGRGQ, seg_pos={0 0}, upload=442304]: Done uploading piece. err: <nil> Dec 26 15:30:38.072975903 (504h1m37.944058948s) [seg_pos={0 0}, upload=442304]: Segment reached optimal threshold of 14 pieces. Dec 26 15:30:38.073453272 (504h1m37.944536317s) [noise=true , address=45.91.116.6:4985 , piece_id=CMQ25ELNYVRW35JY6TM3SV3I4CIPNO5JVE546COPMXSFI52KV4QA, seg_pos={0 0}, upload=442304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNs9MM9jDtNC2gZC63SYsePZTd5mE7dqUm9A5Z5QWdPX8PVnp): protocol: read tcp 188.166.225.170:54480->45.91.116.6:4985: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:30:38.073462369 (504h1m37.944545404s) [noise=true , address=45.91.116.6:4985 , piece_id=CMQ25ELNYVRW35JY6TM3SV3I4CIPNO5JVE546COPMXSFI52KV4QA, seg_pos={0 0}, upload=442304]: Long tail cancellation due to optimal threshold Dec 26 15:30:38.073639436 (504h1m37.944722471s) [noise=true , address=77.90.31.11:4560 , piece_id=XEHGTZEINWBJDVMVEO64UFEWFH2YA2IMO5W5J4B7WUGDLXCJMGVQ, seg_pos={0 0}, upload=442304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH): protocol: expected piece hash; context canceled; EOF Dec 26 15:30:38.073645929 (504h1m37.944728974s) [noise=true , address=77.90.31.11:4560 , piece_id=XEHGTZEINWBJDVMVEO64UFEWFH2YA2IMO5W5J4B7WUGDLXCJMGVQ, seg_pos={0 0}, upload=442304]: Long tail cancellation due to optimal threshold Dec 26 15:30:38.073705773 (504h1m37.944788828s) [noise=true , address=64.49.13.169:4710 , piece_id=W6AZE2H64A6747ZLYAFYGU6MISWQUZIYUJJFNXOMJBW3C4RQR7EQ, seg_pos={0 0}, upload=442304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:36438->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:36438->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36438->64.49.13.169:4710: use of closed network connection Dec 26 15:30:38.073759826 (504h1m37.944842871s) [noise=true , address=64.49.13.169:4710 , piece_id=W6AZE2H64A6747ZLYAFYGU6MISWQUZIYUJJFNXOMJBW3C4RQR7EQ, seg_pos={0 0}, upload=442304]: Long tail cancellation due to optimal threshold Dec 26 15:30:38.073885184 (504h1m37.944968239s) [noise=true , address=77.90.31.17:4803 , piece_id=NDIADSENGEJXVRTVNDJUMHC4W3NIGUUJNPJWDOY5LGEHKSWFDCGQ, seg_pos={0 0}, upload=442304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12onacKUY32faZTFuyEjvMCZA4nVKiRKmTKh1WRFMfif2Kw1Q7Y): protocol: expected piece hash; context canceled; EOF Dec 26 15:30:38.073895354 (504h1m37.944978389s) [noise=true , address=77.90.31.17:4803 , piece_id=NDIADSENGEJXVRTVNDJUMHC4W3NIGUUJNPJWDOY5LGEHKSWFDCGQ, seg_pos={0 0}, upload=442304]: Long tail cancellation due to optimal threshold Dec 26 15:30:38.073923597 (504h1m37.945006632s) [seg_pos={0 0}, upload=442304]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:30:38.074025632 (504h1m37.945108677s) [upload=442304]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:30:38.074032806 (504h1m37.945115851s) [upload=442304]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:30:38.374718093 (504h1m38.245801128s) [upload=442304]: Done uploading segments... Dec 26 15:30:38.374741477 (504h1m38.245824512s) [upload=442304]: Upload finished. err: <nil> Duration: 1m0.002077899s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [..X..XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X.....] duration: 4.917548861s trace: https://jaeger.dev.storj.io/trace/466e826bb589bf71 FAIL (2025-12-26 15:31:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.917575361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.........XX..X.X.....X.....X.X............XXX....XX...X.....] duration: 6.185064663s trace: https://jaeger.dev.storj.io/trace/295f48e94306f825 FAIL (2025-12-26 15:30:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ) context deadline exceeded ======LOGS====== Dec 26 15:26:13.096092554 (503h57m12.967175599s) [upload=442241]: Starting upload Dec 26 15:26:13.096201371 (503h57m12.967284426s) [upload=442241]: Done starting upload Dec 26 15:26:13.096285331 (503h57m12.967368366s) [upload=442241]: Uploading segments... Dec 26 15:26:13.096634736 (503h57m12.967717781s) [upload=442241]: Got next segment. Inline: false Dec 26 15:26:13.096720009 (503h57m12.967803054s) [upload=442241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:26:13.096731480 (503h57m12.967814555s) [upload=442241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:26:13.346386396 (503h57m13.217469431s) [seg_pos={0 0}, upload=442241]: Begin upload segment... Dec 26 15:26:13.346431882 (503h57m13.217514917s) [seg_pos={0 0}, upload=442241]: Done begin upload segment. Dec 26 15:26:13.346455677 (503h57m13.217538723s) [noise=true , address=145.79.227.4:4186 , piece_id=TQBUGQ6SPETEJTLK4AK63X5F5SSZIUU6QKQDVQFV4H25Z2W2V54Q, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.346518798 (503h57m13.217601843s) [noise=true , address=145.79.227.51:4410 , piece_id=CKMYU2WJBXZBNV36LNRODJDVRUWHKX4N7F6QONY3ORA7FOPJZVZQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347012878 (503h57m13.218095924s) [noise=true , address=45.91.116.14:4191 , piece_id=6SUIRACEDO357ODZ5ZYBX6HRXA6KH72M467CEZZTGAHMSXVKHAVQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347077742 (503h57m13.218160787s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347456193 (503h57m13.218539238s) [noise=true , address=45.91.116.5:4299 , piece_id=2GZLZBNFNIZ44QMW3PKNU65WWWUBX5ZSQFUKWGBCIAKDMIYCJS7A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347756103 (503h57m13.218839148s) [noise=true , address=45.91.116.10:4083 , piece_id=5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348078628 (503h57m13.219161703s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348395862 (503h57m13.219478907s) [noise=true , address=145.79.227.15:4876 , piece_id=YOOH7ACHJ2VL3ZJIZY3DL3ZUL42RCEKVFQDFZ22BXDEGLZWBPC5A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348733134 (503h57m13.219816179s) [noise=true , address=77.90.31.18:4808 , piece_id=WAUTGUPSCCXPIV4VENHXEOB57FWC7OQK3GN2UAHHQREX53RJXVAQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.349016083 (503h57m13.220099128s) [noise=true , address=45.91.116.6:4973 , piece_id=KF27ZZMNPB5PRCUMMWSCAVSYRLMNHSO7L2NHCRW3RGKZ7JOI4RFA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.351559577 (503h57m13.222642612s) [noise=true , address=145.79.227.39:4015 , piece_id=BMXFV5TLEC6WJBJFWJBITMAUWVTC3ZU56U7SIYLUTL3GXIEEMCNQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.351893382 (503h57m13.222976417s) [noise=true , address=145.79.227.48:4301 , piece_id=GVZGXRQUAF3TCT5GVQUSI6VT2XX6G5O6PUC4J26REL76VU5ZLALQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.352776163 (503h57m13.223859208s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.353302125 (503h57m13.22438517s) [noise=true , address=77.90.31.15:4700 , piece_id=DV5PGV2FOWNSONKYW5TVO2GEAHRKRFPH4VTQF2FISNKOQ7JFNEKQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.354596349 (503h57m13.225679394s) [noise=true , address=45.91.116.4:4940 , piece_id=CFSW3OH4JGHUTOWS2SR5ZQ6B7TZ5WQT7DGGD3K2ZUMLHEEEDWKHQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.355634927 (503h57m13.226717972s) [noise=true , address=45.91.116.11:4118 , piece_id=V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:15.062260982 (503h57m14.933344028s) [upload=442241]: Next returned nil segment Dec 26 15:26:15.062318943 (503h57m14.933401988s) [upload=442241]: Waiting for error group managing segments... Dec 26 15:26:15.344789265 (503h57m15.2158723s) [noise=true , address=145.79.227.4:4186 , piece_id=TQBUGQ6SPETEJTLK4AK63X5F5SSZIUU6QKQDVQFV4H25Z2W2V54Q, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.348322673 (503h57m15.219405708s) [noise=true , address=145.79.227.15:4876 , piece_id=YOOH7ACHJ2VL3ZJIZY3DL3ZUL42RCEKVFQDFZ22BXDEGLZWBPC5A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.360828747 (503h57m15.231911782s) [noise=true , address=145.79.227.51:4410 , piece_id=CKMYU2WJBXZBNV36LNRODJDVRUWHKX4N7F6QONY3ORA7FOPJZVZQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.370693872 (503h57m15.241776907s) [noise=true , address=145.79.227.39:4015 , piece_id=BMXFV5TLEC6WJBJFWJBITMAUWVTC3ZU56U7SIYLUTL3GXIEEMCNQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.384026740 (503h57m15.255109775s) [noise=true , address=145.79.227.48:4301 , piece_id=GVZGXRQUAF3TCT5GVQUSI6VT2XX6G5O6PUC4J26REL76VU5ZLALQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.604946208 (503h57m15.476029243s) [noise=true , address=45.91.116.5:4299 , piece_id=2GZLZBNFNIZ44QMW3PKNU65WWWUBX5ZSQFUKWGBCIAKDMIYCJS7A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.625370549 (503h57m15.496453584s) [noise=true , address=45.91.116.6:4973 , piece_id=KF27ZZMNPB5PRCUMMWSCAVSYRLMNHSO7L2NHCRW3RGKZ7JOI4RFA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.768372509 (503h57m15.639455544s) [noise=true , address=77.90.31.15:4700 , piece_id=DV5PGV2FOWNSONKYW5TVO2GEAHRKRFPH4VTQF2FISNKOQ7JFNEKQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.772904880 (503h57m15.643987925s) [noise=true , address=45.91.116.4:4940 , piece_id=CFSW3OH4JGHUTOWS2SR5ZQ6B7TZ5WQT7DGGD3K2ZUMLHEEEDWKHQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.776538659 (503h57m15.647621694s) [noise=true , address=45.91.116.11:4118 , piece_id=V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.825341898 (503h57m15.696424933s) [noise=true , address=45.91.116.14:4191 , piece_id=6SUIRACEDO357ODZ5ZYBX6HRXA6KH72M467CEZZTGAHMSXVKHAVQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.209263741 (503h57m16.080346786s) [noise=true , address=77.90.31.18:4808 , piece_id=WAUTGUPSCCXPIV4VENHXEOB57FWC7OQK3GN2UAHHQREX53RJXVAQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.481619854 (503h57m16.352702889s) [noise=true , address=45.91.116.10:4083 , piece_id=5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.481638810 (503h57m16.352721845s) [seg_pos={0 0}, upload=442241]: Segment reached optimal threshold of 13 pieces. Dec 26 15:26:16.481856614 (503h57m16.352939649s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.481863618 (503h57m16.352946653s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482081783 (503h57m16.353164819s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.482087865 (503h57m16.3531709s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482202613 (503h57m16.353285648s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.482208123 (503h57m16.353291169s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482224265 (503h57m16.3533073s) [seg_pos={0 0}, upload=442241]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:26:16.482285512 (503h57m16.353368547s) [upload=442241]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:26:16.482290240 (503h57m16.353373275s) [upload=442241]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:26:16.771425964 (503h57m16.642508999s) [upload=442241]: Done uploading segments... Dec 26 15:26:16.771446753 (503h57m16.642529798s) [upload=442241]: Upload finished. err: <nil> Duration: 6.185091884s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [X..X..XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X....] duration: 3.432818625s trace: https://jaeger.dev.storj.io/trace/466e826bb589bf71 FAIL (2025-12-26 15:30:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.432846429s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X..XX..X.X..X..X.X...XX.XXXX.X..................X..........] duration: 55.984917742s trace: https://jaeger.dev.storj.io/trace/649b436dfcc2315b FAIL (2025-12-26 15:30:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ploading piece... Dec 26 15:20:15.221423190 (503h51m15.092506225s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.221926739 (503h51m15.093009784s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222384111 (503h51m15.093467146s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222766368 (503h51m15.093849403s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222828496 (503h51m15.093911541s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223267943 (503h51m15.094350988s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223409813 (503h51m15.094492858s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223867144 (503h51m15.094950189s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224369451 (503h51m15.095452496s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224780333 (503h51m15.095863368s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225240039 (503h51m15.096323074s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225715373 (503h51m15.096798418s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.226705990 (503h51m15.097789025s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.227259905 (503h51m15.09834295s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:16.563815303 (503h51m16.434898348s) [upload=442149]: Next returned nil segment Dec 26 15:20:16.563835262 (503h51m16.434918337s) [upload=442149]: Waiting for error group managing segments... Dec 26 15:20:16.903963975 (503h51m16.77504702s) [noise=true , address=64.49.13.137:4500 , piece_id=E2KGA6VPKOQLQDONDE2T5PQVI4ENQ36C7BYS2MYWL2GDZ6KCZB5A, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.107984915 (503h51m16.97906795s) [noise=true , address=145.79.227.52:4466 , piece_id=7NVAHOJF72H3DBV5VZKGABRKIDIKTI6TIUCR6RVHHVZNSZA3PELQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.150952764 (503h51m17.022035799s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.179410862 (503h51m17.050493907s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.288743881 (503h51m17.159826927s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.315789470 (503h51m17.186872515s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.426193849 (503h51m17.297276894s) [noise=true , address=45.91.116.8:4074 , piece_id=4FHJLTE4GXQLPIZRZC2XZBM2F6JV35YJ7PEUI7SGAYG5PYZBBBCQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.433610592 (503h51m17.304693627s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.443740401 (503h51m17.314823446s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.990102417 (503h51m17.861185452s) [noise=true , address=145.79.227.44:4227 , piece_id=5QIQ2L6ZYNTQUFC44NHQ2MWLS6Q57UICGLGM2SSK5V7KZU5KR55Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072105685 (503h51m17.94318873s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072663447 (503h51m17.943746482s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072690278 (503h51m17.943773314s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133849817 (503h51m18.004932852s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133871317 (503h51m18.004954352s) [seg_pos={0 0}, upload=442149]: Segment reached optimal threshold of 14 pieces. Dec 26 15:20:18.134067501 (503h51m18.005150546s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 26 15:20:18.134078251 (503h51m18.005161296s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134103530 (503h51m18.005186575s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection Dec 26 15:20:18.134111224 (503h51m18.005194269s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134225823 (503h51m18.005308878s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection Dec 26 15:20:18.134234610 (503h51m18.005317645s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134587341 (503h51m18.005670396s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2): protocol: read tcp 188.166.225.170:57666->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:20:18.134596639 (503h51m18.005679684s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134618430 (503h51m18.005701465s) [seg_pos={0 0}, upload=442149]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:20:18.134714033 (503h51m18.005797078s) [upload=442149]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:20:18.134721266 (503h51m18.005804311s) [upload=442149]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:20:18.446337447 (503h51m18.317420492s) [upload=442149]: Done uploading segments... Dec 26 15:20:18.446364277 (503h51m18.317447322s) [upload=442149]: Upload finished. err: <nil> Duration: 55.984934383s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [XX..X..XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X...] duration: 3.563072357s trace: https://jaeger.dev.storj.io/trace/466e826bb589bf71 FAIL (2025-12-26 15:30:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.563102124s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [XXX..X..XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X..] duration: 5.000449706s trace: https://jaeger.dev.storj.io/trace/466e826bb589bf71 FAIL (2025-12-26 15:29:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000475605s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..........XX..X.X.....X.....X.X............XXX....XX...X....] duration: 5.781192323s trace: https://jaeger.dev.storj.io/trace/295f48e94306f825 FAIL (2025-12-26 15:29:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ) context deadline exceeded ======LOGS====== Dec 26 15:26:13.096092554 (503h57m12.967175599s) [upload=442241]: Starting upload Dec 26 15:26:13.096201371 (503h57m12.967284426s) [upload=442241]: Done starting upload Dec 26 15:26:13.096285331 (503h57m12.967368366s) [upload=442241]: Uploading segments... Dec 26 15:26:13.096634736 (503h57m12.967717781s) [upload=442241]: Got next segment. Inline: false Dec 26 15:26:13.096720009 (503h57m12.967803054s) [upload=442241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:26:13.096731480 (503h57m12.967814555s) [upload=442241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:26:13.346386396 (503h57m13.217469431s) [seg_pos={0 0}, upload=442241]: Begin upload segment... Dec 26 15:26:13.346431882 (503h57m13.217514917s) [seg_pos={0 0}, upload=442241]: Done begin upload segment. Dec 26 15:26:13.346455677 (503h57m13.217538723s) [noise=true , address=145.79.227.4:4186 , piece_id=TQBUGQ6SPETEJTLK4AK63X5F5SSZIUU6QKQDVQFV4H25Z2W2V54Q, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.346518798 (503h57m13.217601843s) [noise=true , address=145.79.227.51:4410 , piece_id=CKMYU2WJBXZBNV36LNRODJDVRUWHKX4N7F6QONY3ORA7FOPJZVZQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347012878 (503h57m13.218095924s) [noise=true , address=45.91.116.14:4191 , piece_id=6SUIRACEDO357ODZ5ZYBX6HRXA6KH72M467CEZZTGAHMSXVKHAVQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347077742 (503h57m13.218160787s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347456193 (503h57m13.218539238s) [noise=true , address=45.91.116.5:4299 , piece_id=2GZLZBNFNIZ44QMW3PKNU65WWWUBX5ZSQFUKWGBCIAKDMIYCJS7A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347756103 (503h57m13.218839148s) [noise=true , address=45.91.116.10:4083 , piece_id=5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348078628 (503h57m13.219161703s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348395862 (503h57m13.219478907s) [noise=true , address=145.79.227.15:4876 , piece_id=YOOH7ACHJ2VL3ZJIZY3DL3ZUL42RCEKVFQDFZ22BXDEGLZWBPC5A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348733134 (503h57m13.219816179s) [noise=true , address=77.90.31.18:4808 , piece_id=WAUTGUPSCCXPIV4VENHXEOB57FWC7OQK3GN2UAHHQREX53RJXVAQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.349016083 (503h57m13.220099128s) [noise=true , address=45.91.116.6:4973 , piece_id=KF27ZZMNPB5PRCUMMWSCAVSYRLMNHSO7L2NHCRW3RGKZ7JOI4RFA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.351559577 (503h57m13.222642612s) [noise=true , address=145.79.227.39:4015 , piece_id=BMXFV5TLEC6WJBJFWJBITMAUWVTC3ZU56U7SIYLUTL3GXIEEMCNQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.351893382 (503h57m13.222976417s) [noise=true , address=145.79.227.48:4301 , piece_id=GVZGXRQUAF3TCT5GVQUSI6VT2XX6G5O6PUC4J26REL76VU5ZLALQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.352776163 (503h57m13.223859208s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.353302125 (503h57m13.22438517s) [noise=true , address=77.90.31.15:4700 , piece_id=DV5PGV2FOWNSONKYW5TVO2GEAHRKRFPH4VTQF2FISNKOQ7JFNEKQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.354596349 (503h57m13.225679394s) [noise=true , address=45.91.116.4:4940 , piece_id=CFSW3OH4JGHUTOWS2SR5ZQ6B7TZ5WQT7DGGD3K2ZUMLHEEEDWKHQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.355634927 (503h57m13.226717972s) [noise=true , address=45.91.116.11:4118 , piece_id=V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:15.062260982 (503h57m14.933344028s) [upload=442241]: Next returned nil segment Dec 26 15:26:15.062318943 (503h57m14.933401988s) [upload=442241]: Waiting for error group managing segments... Dec 26 15:26:15.344789265 (503h57m15.2158723s) [noise=true , address=145.79.227.4:4186 , piece_id=TQBUGQ6SPETEJTLK4AK63X5F5SSZIUU6QKQDVQFV4H25Z2W2V54Q, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.348322673 (503h57m15.219405708s) [noise=true , address=145.79.227.15:4876 , piece_id=YOOH7ACHJ2VL3ZJIZY3DL3ZUL42RCEKVFQDFZ22BXDEGLZWBPC5A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.360828747 (503h57m15.231911782s) [noise=true , address=145.79.227.51:4410 , piece_id=CKMYU2WJBXZBNV36LNRODJDVRUWHKX4N7F6QONY3ORA7FOPJZVZQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.370693872 (503h57m15.241776907s) [noise=true , address=145.79.227.39:4015 , piece_id=BMXFV5TLEC6WJBJFWJBITMAUWVTC3ZU56U7SIYLUTL3GXIEEMCNQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.384026740 (503h57m15.255109775s) [noise=true , address=145.79.227.48:4301 , piece_id=GVZGXRQUAF3TCT5GVQUSI6VT2XX6G5O6PUC4J26REL76VU5ZLALQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.604946208 (503h57m15.476029243s) [noise=true , address=45.91.116.5:4299 , piece_id=2GZLZBNFNIZ44QMW3PKNU65WWWUBX5ZSQFUKWGBCIAKDMIYCJS7A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.625370549 (503h57m15.496453584s) [noise=true , address=45.91.116.6:4973 , piece_id=KF27ZZMNPB5PRCUMMWSCAVSYRLMNHSO7L2NHCRW3RGKZ7JOI4RFA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.768372509 (503h57m15.639455544s) [noise=true , address=77.90.31.15:4700 , piece_id=DV5PGV2FOWNSONKYW5TVO2GEAHRKRFPH4VTQF2FISNKOQ7JFNEKQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.772904880 (503h57m15.643987925s) [noise=true , address=45.91.116.4:4940 , piece_id=CFSW3OH4JGHUTOWS2SR5ZQ6B7TZ5WQT7DGGD3K2ZUMLHEEEDWKHQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.776538659 (503h57m15.647621694s) [noise=true , address=45.91.116.11:4118 , piece_id=V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.825341898 (503h57m15.696424933s) [noise=true , address=45.91.116.14:4191 , piece_id=6SUIRACEDO357ODZ5ZYBX6HRXA6KH72M467CEZZTGAHMSXVKHAVQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.209263741 (503h57m16.080346786s) [noise=true , address=77.90.31.18:4808 , piece_id=WAUTGUPSCCXPIV4VENHXEOB57FWC7OQK3GN2UAHHQREX53RJXVAQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.481619854 (503h57m16.352702889s) [noise=true , address=45.91.116.10:4083 , piece_id=5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.481638810 (503h57m16.352721845s) [seg_pos={0 0}, upload=442241]: Segment reached optimal threshold of 13 pieces. Dec 26 15:26:16.481856614 (503h57m16.352939649s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.481863618 (503h57m16.352946653s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482081783 (503h57m16.353164819s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.482087865 (503h57m16.3531709s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482202613 (503h57m16.353285648s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.482208123 (503h57m16.353291169s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482224265 (503h57m16.3533073s) [seg_pos={0 0}, upload=442241]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:26:16.482285512 (503h57m16.353368547s) [upload=442241]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:26:16.482290240 (503h57m16.353373275s) [upload=442241]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:26:16.771425964 (503h57m16.642508999s) [upload=442241]: Done uploading segments... Dec 26 15:26:16.771446753 (503h57m16.642529798s) [upload=442241]: Upload finished. err: <nil> Duration: 5.781218294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X..XX..X.X..X..X.X...XX.XXXX.X..................X.........] duration: 53.268827689s trace: https://jaeger.dev.storj.io/trace/649b436dfcc2315b FAIL (2025-12-26 15:29:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ploading piece... Dec 26 15:20:15.221423190 (503h51m15.092506225s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.221926739 (503h51m15.093009784s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222384111 (503h51m15.093467146s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222766368 (503h51m15.093849403s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222828496 (503h51m15.093911541s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223267943 (503h51m15.094350988s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223409813 (503h51m15.094492858s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223867144 (503h51m15.094950189s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224369451 (503h51m15.095452496s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224780333 (503h51m15.095863368s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225240039 (503h51m15.096323074s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225715373 (503h51m15.096798418s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.226705990 (503h51m15.097789025s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.227259905 (503h51m15.09834295s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:16.563815303 (503h51m16.434898348s) [upload=442149]: Next returned nil segment Dec 26 15:20:16.563835262 (503h51m16.434918337s) [upload=442149]: Waiting for error group managing segments... Dec 26 15:20:16.903963975 (503h51m16.77504702s) [noise=true , address=64.49.13.137:4500 , piece_id=E2KGA6VPKOQLQDONDE2T5PQVI4ENQ36C7BYS2MYWL2GDZ6KCZB5A, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.107984915 (503h51m16.97906795s) [noise=true , address=145.79.227.52:4466 , piece_id=7NVAHOJF72H3DBV5VZKGABRKIDIKTI6TIUCR6RVHHVZNSZA3PELQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.150952764 (503h51m17.022035799s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.179410862 (503h51m17.050493907s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.288743881 (503h51m17.159826927s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.315789470 (503h51m17.186872515s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.426193849 (503h51m17.297276894s) [noise=true , address=45.91.116.8:4074 , piece_id=4FHJLTE4GXQLPIZRZC2XZBM2F6JV35YJ7PEUI7SGAYG5PYZBBBCQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.433610592 (503h51m17.304693627s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.443740401 (503h51m17.314823446s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.990102417 (503h51m17.861185452s) [noise=true , address=145.79.227.44:4227 , piece_id=5QIQ2L6ZYNTQUFC44NHQ2MWLS6Q57UICGLGM2SSK5V7KZU5KR55Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072105685 (503h51m17.94318873s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072663447 (503h51m17.943746482s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072690278 (503h51m17.943773314s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133849817 (503h51m18.004932852s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133871317 (503h51m18.004954352s) [seg_pos={0 0}, upload=442149]: Segment reached optimal threshold of 14 pieces. Dec 26 15:20:18.134067501 (503h51m18.005150546s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 26 15:20:18.134078251 (503h51m18.005161296s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134103530 (503h51m18.005186575s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection Dec 26 15:20:18.134111224 (503h51m18.005194269s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134225823 (503h51m18.005308878s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection Dec 26 15:20:18.134234610 (503h51m18.005317645s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134587341 (503h51m18.005670396s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2): protocol: read tcp 188.166.225.170:57666->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:20:18.134596639 (503h51m18.005679684s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134618430 (503h51m18.005701465s) [seg_pos={0 0}, upload=442149]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:20:18.134714033 (503h51m18.005797078s) [upload=442149]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:20:18.134721266 (503h51m18.005804311s) [upload=442149]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:20:18.446337447 (503h51m18.317420492s) [upload=442149]: Done uploading segments... Dec 26 15:20:18.446364277 (503h51m18.317447322s) [upload=442149]: Upload finished. err: <nil> Duration: 53.268846175s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...........XX..X.X.....X.....X.X............XXX....XX...X...] duration: 6.054178968s trace: https://jaeger.dev.storj.io/trace/295f48e94306f825 FAIL (2025-12-26 15:29:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ) context deadline exceeded ======LOGS====== Dec 26 15:26:13.096092554 (503h57m12.967175599s) [upload=442241]: Starting upload Dec 26 15:26:13.096201371 (503h57m12.967284426s) [upload=442241]: Done starting upload Dec 26 15:26:13.096285331 (503h57m12.967368366s) [upload=442241]: Uploading segments... Dec 26 15:26:13.096634736 (503h57m12.967717781s) [upload=442241]: Got next segment. Inline: false Dec 26 15:26:13.096720009 (503h57m12.967803054s) [upload=442241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:26:13.096731480 (503h57m12.967814555s) [upload=442241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:26:13.346386396 (503h57m13.217469431s) [seg_pos={0 0}, upload=442241]: Begin upload segment... Dec 26 15:26:13.346431882 (503h57m13.217514917s) [seg_pos={0 0}, upload=442241]: Done begin upload segment. Dec 26 15:26:13.346455677 (503h57m13.217538723s) [noise=true , address=145.79.227.4:4186 , piece_id=TQBUGQ6SPETEJTLK4AK63X5F5SSZIUU6QKQDVQFV4H25Z2W2V54Q, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.346518798 (503h57m13.217601843s) [noise=true , address=145.79.227.51:4410 , piece_id=CKMYU2WJBXZBNV36LNRODJDVRUWHKX4N7F6QONY3ORA7FOPJZVZQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347012878 (503h57m13.218095924s) [noise=true , address=45.91.116.14:4191 , piece_id=6SUIRACEDO357ODZ5ZYBX6HRXA6KH72M467CEZZTGAHMSXVKHAVQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347077742 (503h57m13.218160787s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347456193 (503h57m13.218539238s) [noise=true , address=45.91.116.5:4299 , piece_id=2GZLZBNFNIZ44QMW3PKNU65WWWUBX5ZSQFUKWGBCIAKDMIYCJS7A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347756103 (503h57m13.218839148s) [noise=true , address=45.91.116.10:4083 , piece_id=5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348078628 (503h57m13.219161703s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348395862 (503h57m13.219478907s) [noise=true , address=145.79.227.15:4876 , piece_id=YOOH7ACHJ2VL3ZJIZY3DL3ZUL42RCEKVFQDFZ22BXDEGLZWBPC5A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348733134 (503h57m13.219816179s) [noise=true , address=77.90.31.18:4808 , piece_id=WAUTGUPSCCXPIV4VENHXEOB57FWC7OQK3GN2UAHHQREX53RJXVAQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.349016083 (503h57m13.220099128s) [noise=true , address=45.91.116.6:4973 , piece_id=KF27ZZMNPB5PRCUMMWSCAVSYRLMNHSO7L2NHCRW3RGKZ7JOI4RFA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.351559577 (503h57m13.222642612s) [noise=true , address=145.79.227.39:4015 , piece_id=BMXFV5TLEC6WJBJFWJBITMAUWVTC3ZU56U7SIYLUTL3GXIEEMCNQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.351893382 (503h57m13.222976417s) [noise=true , address=145.79.227.48:4301 , piece_id=GVZGXRQUAF3TCT5GVQUSI6VT2XX6G5O6PUC4J26REL76VU5ZLALQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.352776163 (503h57m13.223859208s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.353302125 (503h57m13.22438517s) [noise=true , address=77.90.31.15:4700 , piece_id=DV5PGV2FOWNSONKYW5TVO2GEAHRKRFPH4VTQF2FISNKOQ7JFNEKQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.354596349 (503h57m13.225679394s) [noise=true , address=45.91.116.4:4940 , piece_id=CFSW3OH4JGHUTOWS2SR5ZQ6B7TZ5WQT7DGGD3K2ZUMLHEEEDWKHQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.355634927 (503h57m13.226717972s) [noise=true , address=45.91.116.11:4118 , piece_id=V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:15.062260982 (503h57m14.933344028s) [upload=442241]: Next returned nil segment Dec 26 15:26:15.062318943 (503h57m14.933401988s) [upload=442241]: Waiting for error group managing segments... Dec 26 15:26:15.344789265 (503h57m15.2158723s) [noise=true , address=145.79.227.4:4186 , piece_id=TQBUGQ6SPETEJTLK4AK63X5F5SSZIUU6QKQDVQFV4H25Z2W2V54Q, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.348322673 (503h57m15.219405708s) [noise=true , address=145.79.227.15:4876 , piece_id=YOOH7ACHJ2VL3ZJIZY3DL3ZUL42RCEKVFQDFZ22BXDEGLZWBPC5A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.360828747 (503h57m15.231911782s) [noise=true , address=145.79.227.51:4410 , piece_id=CKMYU2WJBXZBNV36LNRODJDVRUWHKX4N7F6QONY3ORA7FOPJZVZQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.370693872 (503h57m15.241776907s) [noise=true , address=145.79.227.39:4015 , piece_id=BMXFV5TLEC6WJBJFWJBITMAUWVTC3ZU56U7SIYLUTL3GXIEEMCNQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.384026740 (503h57m15.255109775s) [noise=true , address=145.79.227.48:4301 , piece_id=GVZGXRQUAF3TCT5GVQUSI6VT2XX6G5O6PUC4J26REL76VU5ZLALQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.604946208 (503h57m15.476029243s) [noise=true , address=45.91.116.5:4299 , piece_id=2GZLZBNFNIZ44QMW3PKNU65WWWUBX5ZSQFUKWGBCIAKDMIYCJS7A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.625370549 (503h57m15.496453584s) [noise=true , address=45.91.116.6:4973 , piece_id=KF27ZZMNPB5PRCUMMWSCAVSYRLMNHSO7L2NHCRW3RGKZ7JOI4RFA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.768372509 (503h57m15.639455544s) [noise=true , address=77.90.31.15:4700 , piece_id=DV5PGV2FOWNSONKYW5TVO2GEAHRKRFPH4VTQF2FISNKOQ7JFNEKQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.772904880 (503h57m15.643987925s) [noise=true , address=45.91.116.4:4940 , piece_id=CFSW3OH4JGHUTOWS2SR5ZQ6B7TZ5WQT7DGGD3K2ZUMLHEEEDWKHQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.776538659 (503h57m15.647621694s) [noise=true , address=45.91.116.11:4118 , piece_id=V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.825341898 (503h57m15.696424933s) [noise=true , address=45.91.116.14:4191 , piece_id=6SUIRACEDO357ODZ5ZYBX6HRXA6KH72M467CEZZTGAHMSXVKHAVQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.209263741 (503h57m16.080346786s) [noise=true , address=77.90.31.18:4808 , piece_id=WAUTGUPSCCXPIV4VENHXEOB57FWC7OQK3GN2UAHHQREX53RJXVAQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.481619854 (503h57m16.352702889s) [noise=true , address=45.91.116.10:4083 , piece_id=5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.481638810 (503h57m16.352721845s) [seg_pos={0 0}, upload=442241]: Segment reached optimal threshold of 13 pieces. Dec 26 15:26:16.481856614 (503h57m16.352939649s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.481863618 (503h57m16.352946653s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482081783 (503h57m16.353164819s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.482087865 (503h57m16.3531709s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482202613 (503h57m16.353285648s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.482208123 (503h57m16.353291169s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482224265 (503h57m16.3533073s) [seg_pos={0 0}, upload=442241]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:26:16.482285512 (503h57m16.353368547s) [upload=442241]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:26:16.482290240 (503h57m16.353373275s) [upload=442241]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:26:16.771425964 (503h57m16.642508999s) [upload=442241]: Done uploading segments... Dec 26 15:26:16.771446753 (503h57m16.642529798s) [upload=442241]: Upload finished. err: <nil> Duration: 6.054206459s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [XXXX..X..XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X.] duration: 3.830978336s trace: https://jaeger.dev.storj.io/trace/466e826bb589bf71 FAIL (2025-12-26 15:28:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.831008344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [XX.X..XX..X.X..X..X.X...XX.XXXX.X..................X........] duration: 42.867636301s trace: https://jaeger.dev.storj.io/trace/649b436dfcc2315b FAIL (2025-12-26 15:28:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ploading piece... Dec 26 15:20:15.221423190 (503h51m15.092506225s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.221926739 (503h51m15.093009784s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222384111 (503h51m15.093467146s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222766368 (503h51m15.093849403s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222828496 (503h51m15.093911541s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223267943 (503h51m15.094350988s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223409813 (503h51m15.094492858s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223867144 (503h51m15.094950189s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224369451 (503h51m15.095452496s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224780333 (503h51m15.095863368s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225240039 (503h51m15.096323074s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225715373 (503h51m15.096798418s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.226705990 (503h51m15.097789025s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.227259905 (503h51m15.09834295s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:16.563815303 (503h51m16.434898348s) [upload=442149]: Next returned nil segment Dec 26 15:20:16.563835262 (503h51m16.434918337s) [upload=442149]: Waiting for error group managing segments... Dec 26 15:20:16.903963975 (503h51m16.77504702s) [noise=true , address=64.49.13.137:4500 , piece_id=E2KGA6VPKOQLQDONDE2T5PQVI4ENQ36C7BYS2MYWL2GDZ6KCZB5A, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.107984915 (503h51m16.97906795s) [noise=true , address=145.79.227.52:4466 , piece_id=7NVAHOJF72H3DBV5VZKGABRKIDIKTI6TIUCR6RVHHVZNSZA3PELQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.150952764 (503h51m17.022035799s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.179410862 (503h51m17.050493907s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.288743881 (503h51m17.159826927s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.315789470 (503h51m17.186872515s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.426193849 (503h51m17.297276894s) [noise=true , address=45.91.116.8:4074 , piece_id=4FHJLTE4GXQLPIZRZC2XZBM2F6JV35YJ7PEUI7SGAYG5PYZBBBCQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.433610592 (503h51m17.304693627s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.443740401 (503h51m17.314823446s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.990102417 (503h51m17.861185452s) [noise=true , address=145.79.227.44:4227 , piece_id=5QIQ2L6ZYNTQUFC44NHQ2MWLS6Q57UICGLGM2SSK5V7KZU5KR55Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072105685 (503h51m17.94318873s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072663447 (503h51m17.943746482s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072690278 (503h51m17.943773314s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133849817 (503h51m18.004932852s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133871317 (503h51m18.004954352s) [seg_pos={0 0}, upload=442149]: Segment reached optimal threshold of 14 pieces. Dec 26 15:20:18.134067501 (503h51m18.005150546s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 26 15:20:18.134078251 (503h51m18.005161296s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134103530 (503h51m18.005186575s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection Dec 26 15:20:18.134111224 (503h51m18.005194269s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134225823 (503h51m18.005308878s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection Dec 26 15:20:18.134234610 (503h51m18.005317645s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134587341 (503h51m18.005670396s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2): protocol: read tcp 188.166.225.170:57666->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:20:18.134596639 (503h51m18.005679684s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134618430 (503h51m18.005701465s) [seg_pos={0 0}, upload=442149]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:20:18.134714033 (503h51m18.005797078s) [upload=442149]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:20:18.134721266 (503h51m18.005804311s) [upload=442149]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:20:18.446337447 (503h51m18.317420492s) [upload=442149]: Done uploading segments... Dec 26 15:20:18.446364277 (503h51m18.317447322s) [upload=442149]: Upload finished. err: <nil> Duration: 42.867652752s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X...........XX..X.X.....X.....X.X............XXX....XX...X..] duration: 6.127983303s trace: https://jaeger.dev.storj.io/trace/295f48e94306f825 FAIL (2025-12-26 15:27:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ) context deadline exceeded ======LOGS====== Dec 26 15:26:13.096092554 (503h57m12.967175599s) [upload=442241]: Starting upload Dec 26 15:26:13.096201371 (503h57m12.967284426s) [upload=442241]: Done starting upload Dec 26 15:26:13.096285331 (503h57m12.967368366s) [upload=442241]: Uploading segments... Dec 26 15:26:13.096634736 (503h57m12.967717781s) [upload=442241]: Got next segment. Inline: false Dec 26 15:26:13.096720009 (503h57m12.967803054s) [upload=442241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:26:13.096731480 (503h57m12.967814555s) [upload=442241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:26:13.346386396 (503h57m13.217469431s) [seg_pos={0 0}, upload=442241]: Begin upload segment... Dec 26 15:26:13.346431882 (503h57m13.217514917s) [seg_pos={0 0}, upload=442241]: Done begin upload segment. Dec 26 15:26:13.346455677 (503h57m13.217538723s) [noise=true , address=145.79.227.4:4186 , piece_id=TQBUGQ6SPETEJTLK4AK63X5F5SSZIUU6QKQDVQFV4H25Z2W2V54Q, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.346518798 (503h57m13.217601843s) [noise=true , address=145.79.227.51:4410 , piece_id=CKMYU2WJBXZBNV36LNRODJDVRUWHKX4N7F6QONY3ORA7FOPJZVZQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347012878 (503h57m13.218095924s) [noise=true , address=45.91.116.14:4191 , piece_id=6SUIRACEDO357ODZ5ZYBX6HRXA6KH72M467CEZZTGAHMSXVKHAVQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347077742 (503h57m13.218160787s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347456193 (503h57m13.218539238s) [noise=true , address=45.91.116.5:4299 , piece_id=2GZLZBNFNIZ44QMW3PKNU65WWWUBX5ZSQFUKWGBCIAKDMIYCJS7A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347756103 (503h57m13.218839148s) [noise=true , address=45.91.116.10:4083 , piece_id=5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348078628 (503h57m13.219161703s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348395862 (503h57m13.219478907s) [noise=true , address=145.79.227.15:4876 , piece_id=YOOH7ACHJ2VL3ZJIZY3DL3ZUL42RCEKVFQDFZ22BXDEGLZWBPC5A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348733134 (503h57m13.219816179s) [noise=true , address=77.90.31.18:4808 , piece_id=WAUTGUPSCCXPIV4VENHXEOB57FWC7OQK3GN2UAHHQREX53RJXVAQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.349016083 (503h57m13.220099128s) [noise=true , address=45.91.116.6:4973 , piece_id=KF27ZZMNPB5PRCUMMWSCAVSYRLMNHSO7L2NHCRW3RGKZ7JOI4RFA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.351559577 (503h57m13.222642612s) [noise=true , address=145.79.227.39:4015 , piece_id=BMXFV5TLEC6WJBJFWJBITMAUWVTC3ZU56U7SIYLUTL3GXIEEMCNQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.351893382 (503h57m13.222976417s) [noise=true , address=145.79.227.48:4301 , piece_id=GVZGXRQUAF3TCT5GVQUSI6VT2XX6G5O6PUC4J26REL76VU5ZLALQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.352776163 (503h57m13.223859208s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.353302125 (503h57m13.22438517s) [noise=true , address=77.90.31.15:4700 , piece_id=DV5PGV2FOWNSONKYW5TVO2GEAHRKRFPH4VTQF2FISNKOQ7JFNEKQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.354596349 (503h57m13.225679394s) [noise=true , address=45.91.116.4:4940 , piece_id=CFSW3OH4JGHUTOWS2SR5ZQ6B7TZ5WQT7DGGD3K2ZUMLHEEEDWKHQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.355634927 (503h57m13.226717972s) [noise=true , address=45.91.116.11:4118 , piece_id=V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:15.062260982 (503h57m14.933344028s) [upload=442241]: Next returned nil segment Dec 26 15:26:15.062318943 (503h57m14.933401988s) [upload=442241]: Waiting for error group managing segments... Dec 26 15:26:15.344789265 (503h57m15.2158723s) [noise=true , address=145.79.227.4:4186 , piece_id=TQBUGQ6SPETEJTLK4AK63X5F5SSZIUU6QKQDVQFV4H25Z2W2V54Q, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.348322673 (503h57m15.219405708s) [noise=true , address=145.79.227.15:4876 , piece_id=YOOH7ACHJ2VL3ZJIZY3DL3ZUL42RCEKVFQDFZ22BXDEGLZWBPC5A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.360828747 (503h57m15.231911782s) [noise=true , address=145.79.227.51:4410 , piece_id=CKMYU2WJBXZBNV36LNRODJDVRUWHKX4N7F6QONY3ORA7FOPJZVZQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.370693872 (503h57m15.241776907s) [noise=true , address=145.79.227.39:4015 , piece_id=BMXFV5TLEC6WJBJFWJBITMAUWVTC3ZU56U7SIYLUTL3GXIEEMCNQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.384026740 (503h57m15.255109775s) [noise=true , address=145.79.227.48:4301 , piece_id=GVZGXRQUAF3TCT5GVQUSI6VT2XX6G5O6PUC4J26REL76VU5ZLALQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.604946208 (503h57m15.476029243s) [noise=true , address=45.91.116.5:4299 , piece_id=2GZLZBNFNIZ44QMW3PKNU65WWWUBX5ZSQFUKWGBCIAKDMIYCJS7A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.625370549 (503h57m15.496453584s) [noise=true , address=45.91.116.6:4973 , piece_id=KF27ZZMNPB5PRCUMMWSCAVSYRLMNHSO7L2NHCRW3RGKZ7JOI4RFA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.768372509 (503h57m15.639455544s) [noise=true , address=77.90.31.15:4700 , piece_id=DV5PGV2FOWNSONKYW5TVO2GEAHRKRFPH4VTQF2FISNKOQ7JFNEKQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.772904880 (503h57m15.643987925s) [noise=true , address=45.91.116.4:4940 , piece_id=CFSW3OH4JGHUTOWS2SR5ZQ6B7TZ5WQT7DGGD3K2ZUMLHEEEDWKHQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.776538659 (503h57m15.647621694s) [noise=true , address=45.91.116.11:4118 , piece_id=V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.825341898 (503h57m15.696424933s) [noise=true , address=45.91.116.14:4191 , piece_id=6SUIRACEDO357ODZ5ZYBX6HRXA6KH72M467CEZZTGAHMSXVKHAVQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.209263741 (503h57m16.080346786s) [noise=true , address=77.90.31.18:4808 , piece_id=WAUTGUPSCCXPIV4VENHXEOB57FWC7OQK3GN2UAHHQREX53RJXVAQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.481619854 (503h57m16.352702889s) [noise=true , address=45.91.116.10:4083 , piece_id=5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.481638810 (503h57m16.352721845s) [seg_pos={0 0}, upload=442241]: Segment reached optimal threshold of 13 pieces. Dec 26 15:26:16.481856614 (503h57m16.352939649s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.481863618 (503h57m16.352946653s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482081783 (503h57m16.353164819s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.482087865 (503h57m16.3531709s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482202613 (503h57m16.353285648s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.482208123 (503h57m16.353291169s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482224265 (503h57m16.3533073s) [seg_pos={0 0}, upload=442241]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:26:16.482285512 (503h57m16.353368547s) [upload=442241]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:26:16.482290240 (503h57m16.353373275s) [upload=442241]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:26:16.771425964 (503h57m16.642508999s) [upload=442241]: Done uploading segments... Dec 26 15:26:16.771446753 (503h57m16.642529798s) [upload=442241]: Upload finished. err: <nil> Duration: 6.128010474s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [XXXXX..X..XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...X] duration: 5.001206128s trace: https://jaeger.dev.storj.io/trace/466e826bb589bf71 FAIL (2025-12-26 15:27:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001230384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [.XX.X..XX..X.X..X..X.X...XX.XXXX.X..................X.......] duration: 48.452073963s trace: https://jaeger.dev.storj.io/trace/649b436dfcc2315b FAIL (2025-12-26 15:27:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ploading piece... Dec 26 15:20:15.221423190 (503h51m15.092506225s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.221926739 (503h51m15.093009784s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222384111 (503h51m15.093467146s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222766368 (503h51m15.093849403s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222828496 (503h51m15.093911541s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223267943 (503h51m15.094350988s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223409813 (503h51m15.094492858s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223867144 (503h51m15.094950189s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224369451 (503h51m15.095452496s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224780333 (503h51m15.095863368s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225240039 (503h51m15.096323074s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225715373 (503h51m15.096798418s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.226705990 (503h51m15.097789025s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.227259905 (503h51m15.09834295s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:16.563815303 (503h51m16.434898348s) [upload=442149]: Next returned nil segment Dec 26 15:20:16.563835262 (503h51m16.434918337s) [upload=442149]: Waiting for error group managing segments... Dec 26 15:20:16.903963975 (503h51m16.77504702s) [noise=true , address=64.49.13.137:4500 , piece_id=E2KGA6VPKOQLQDONDE2T5PQVI4ENQ36C7BYS2MYWL2GDZ6KCZB5A, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.107984915 (503h51m16.97906795s) [noise=true , address=145.79.227.52:4466 , piece_id=7NVAHOJF72H3DBV5VZKGABRKIDIKTI6TIUCR6RVHHVZNSZA3PELQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.150952764 (503h51m17.022035799s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.179410862 (503h51m17.050493907s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.288743881 (503h51m17.159826927s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.315789470 (503h51m17.186872515s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.426193849 (503h51m17.297276894s) [noise=true , address=45.91.116.8:4074 , piece_id=4FHJLTE4GXQLPIZRZC2XZBM2F6JV35YJ7PEUI7SGAYG5PYZBBBCQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.433610592 (503h51m17.304693627s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.443740401 (503h51m17.314823446s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.990102417 (503h51m17.861185452s) [noise=true , address=145.79.227.44:4227 , piece_id=5QIQ2L6ZYNTQUFC44NHQ2MWLS6Q57UICGLGM2SSK5V7KZU5KR55Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072105685 (503h51m17.94318873s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072663447 (503h51m17.943746482s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072690278 (503h51m17.943773314s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133849817 (503h51m18.004932852s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133871317 (503h51m18.004954352s) [seg_pos={0 0}, upload=442149]: Segment reached optimal threshold of 14 pieces. Dec 26 15:20:18.134067501 (503h51m18.005150546s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 26 15:20:18.134078251 (503h51m18.005161296s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134103530 (503h51m18.005186575s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection Dec 26 15:20:18.134111224 (503h51m18.005194269s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134225823 (503h51m18.005308878s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection Dec 26 15:20:18.134234610 (503h51m18.005317645s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134587341 (503h51m18.005670396s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2): protocol: read tcp 188.166.225.170:57666->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:20:18.134596639 (503h51m18.005679684s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134618430 (503h51m18.005701465s) [seg_pos={0 0}, upload=442149]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:20:18.134714033 (503h51m18.005797078s) [upload=442149]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:20:18.134721266 (503h51m18.005804311s) [upload=442149]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:20:18.446337447 (503h51m18.317420492s) [upload=442149]: Done uploading segments... Dec 26 15:20:18.446364277 (503h51m18.317447322s) [upload=442149]: Upload finished. err: <nil> Duration: 48.452108819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XX...........XX..X.X.....X.....X.X............XXX....XX...X.] duration: 6.022265618s trace: https://jaeger.dev.storj.io/trace/295f48e94306f825 FAIL (2025-12-26 15:27:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ) context deadline exceeded ======LOGS====== Dec 26 15:26:13.096092554 (503h57m12.967175599s) [upload=442241]: Starting upload Dec 26 15:26:13.096201371 (503h57m12.967284426s) [upload=442241]: Done starting upload Dec 26 15:26:13.096285331 (503h57m12.967368366s) [upload=442241]: Uploading segments... Dec 26 15:26:13.096634736 (503h57m12.967717781s) [upload=442241]: Got next segment. Inline: false Dec 26 15:26:13.096720009 (503h57m12.967803054s) [upload=442241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:26:13.096731480 (503h57m12.967814555s) [upload=442241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:26:13.346386396 (503h57m13.217469431s) [seg_pos={0 0}, upload=442241]: Begin upload segment... Dec 26 15:26:13.346431882 (503h57m13.217514917s) [seg_pos={0 0}, upload=442241]: Done begin upload segment. Dec 26 15:26:13.346455677 (503h57m13.217538723s) [noise=true , address=145.79.227.4:4186 , piece_id=TQBUGQ6SPETEJTLK4AK63X5F5SSZIUU6QKQDVQFV4H25Z2W2V54Q, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.346518798 (503h57m13.217601843s) [noise=true , address=145.79.227.51:4410 , piece_id=CKMYU2WJBXZBNV36LNRODJDVRUWHKX4N7F6QONY3ORA7FOPJZVZQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347012878 (503h57m13.218095924s) [noise=true , address=45.91.116.14:4191 , piece_id=6SUIRACEDO357ODZ5ZYBX6HRXA6KH72M467CEZZTGAHMSXVKHAVQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347077742 (503h57m13.218160787s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347456193 (503h57m13.218539238s) [noise=true , address=45.91.116.5:4299 , piece_id=2GZLZBNFNIZ44QMW3PKNU65WWWUBX5ZSQFUKWGBCIAKDMIYCJS7A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347756103 (503h57m13.218839148s) [noise=true , address=45.91.116.10:4083 , piece_id=5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348078628 (503h57m13.219161703s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348395862 (503h57m13.219478907s) [noise=true , address=145.79.227.15:4876 , piece_id=YOOH7ACHJ2VL3ZJIZY3DL3ZUL42RCEKVFQDFZ22BXDEGLZWBPC5A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348733134 (503h57m13.219816179s) [noise=true , address=77.90.31.18:4808 , piece_id=WAUTGUPSCCXPIV4VENHXEOB57FWC7OQK3GN2UAHHQREX53RJXVAQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.349016083 (503h57m13.220099128s) [noise=true , address=45.91.116.6:4973 , piece_id=KF27ZZMNPB5PRCUMMWSCAVSYRLMNHSO7L2NHCRW3RGKZ7JOI4RFA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.351559577 (503h57m13.222642612s) [noise=true , address=145.79.227.39:4015 , piece_id=BMXFV5TLEC6WJBJFWJBITMAUWVTC3ZU56U7SIYLUTL3GXIEEMCNQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.351893382 (503h57m13.222976417s) [noise=true , address=145.79.227.48:4301 , piece_id=GVZGXRQUAF3TCT5GVQUSI6VT2XX6G5O6PUC4J26REL76VU5ZLALQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.352776163 (503h57m13.223859208s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.353302125 (503h57m13.22438517s) [noise=true , address=77.90.31.15:4700 , piece_id=DV5PGV2FOWNSONKYW5TVO2GEAHRKRFPH4VTQF2FISNKOQ7JFNEKQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.354596349 (503h57m13.225679394s) [noise=true , address=45.91.116.4:4940 , piece_id=CFSW3OH4JGHUTOWS2SR5ZQ6B7TZ5WQT7DGGD3K2ZUMLHEEEDWKHQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.355634927 (503h57m13.226717972s) [noise=true , address=45.91.116.11:4118 , piece_id=V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:15.062260982 (503h57m14.933344028s) [upload=442241]: Next returned nil segment Dec 26 15:26:15.062318943 (503h57m14.933401988s) [upload=442241]: Waiting for error group managing segments... Dec 26 15:26:15.344789265 (503h57m15.2158723s) [noise=true , address=145.79.227.4:4186 , piece_id=TQBUGQ6SPETEJTLK4AK63X5F5SSZIUU6QKQDVQFV4H25Z2W2V54Q, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.348322673 (503h57m15.219405708s) [noise=true , address=145.79.227.15:4876 , piece_id=YOOH7ACHJ2VL3ZJIZY3DL3ZUL42RCEKVFQDFZ22BXDEGLZWBPC5A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.360828747 (503h57m15.231911782s) [noise=true , address=145.79.227.51:4410 , piece_id=CKMYU2WJBXZBNV36LNRODJDVRUWHKX4N7F6QONY3ORA7FOPJZVZQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.370693872 (503h57m15.241776907s) [noise=true , address=145.79.227.39:4015 , piece_id=BMXFV5TLEC6WJBJFWJBITMAUWVTC3ZU56U7SIYLUTL3GXIEEMCNQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.384026740 (503h57m15.255109775s) [noise=true , address=145.79.227.48:4301 , piece_id=GVZGXRQUAF3TCT5GVQUSI6VT2XX6G5O6PUC4J26REL76VU5ZLALQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.604946208 (503h57m15.476029243s) [noise=true , address=45.91.116.5:4299 , piece_id=2GZLZBNFNIZ44QMW3PKNU65WWWUBX5ZSQFUKWGBCIAKDMIYCJS7A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.625370549 (503h57m15.496453584s) [noise=true , address=45.91.116.6:4973 , piece_id=KF27ZZMNPB5PRCUMMWSCAVSYRLMNHSO7L2NHCRW3RGKZ7JOI4RFA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.768372509 (503h57m15.639455544s) [noise=true , address=77.90.31.15:4700 , piece_id=DV5PGV2FOWNSONKYW5TVO2GEAHRKRFPH4VTQF2FISNKOQ7JFNEKQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.772904880 (503h57m15.643987925s) [noise=true , address=45.91.116.4:4940 , piece_id=CFSW3OH4JGHUTOWS2SR5ZQ6B7TZ5WQT7DGGD3K2ZUMLHEEEDWKHQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.776538659 (503h57m15.647621694s) [noise=true , address=45.91.116.11:4118 , piece_id=V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.825341898 (503h57m15.696424933s) [noise=true , address=45.91.116.14:4191 , piece_id=6SUIRACEDO357ODZ5ZYBX6HRXA6KH72M467CEZZTGAHMSXVKHAVQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.209263741 (503h57m16.080346786s) [noise=true , address=77.90.31.18:4808 , piece_id=WAUTGUPSCCXPIV4VENHXEOB57FWC7OQK3GN2UAHHQREX53RJXVAQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.481619854 (503h57m16.352702889s) [noise=true , address=45.91.116.10:4083 , piece_id=5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.481638810 (503h57m16.352721845s) [seg_pos={0 0}, upload=442241]: Segment reached optimal threshold of 13 pieces. Dec 26 15:26:16.481856614 (503h57m16.352939649s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.481863618 (503h57m16.352946653s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482081783 (503h57m16.353164819s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.482087865 (503h57m16.3531709s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482202613 (503h57m16.353285648s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.482208123 (503h57m16.353291169s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482224265 (503h57m16.3533073s) [seg_pos={0 0}, upload=442241]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:26:16.482285512 (503h57m16.353368547s) [upload=442241]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:26:16.482290240 (503h57m16.353373275s) [upload=442241]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:26:16.771425964 (503h57m16.642508999s) [upload=442241]: Done uploading segments... Dec 26 15:26:16.771446753 (503h57m16.642529798s) [upload=442241]: Upload finished. err: <nil> Duration: 6.022350428s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [XXXXXX..X..XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX...] duration: 5.000923258s trace: https://jaeger.dev.storj.io/trace/5c8ebbc50ab9d703 FAIL (2025-12-26 15:26:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000944749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XX...........XX..X.X.....X.....X.X............XXX....XX...X] duration: 30.003012929s trace: https://jaeger.dev.storj.io/trace/295f48e94306f825 FAIL (2025-12-26 15:26:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ) context deadline exceeded ======LOGS====== Dec 26 15:26:13.096092554 (503h57m12.967175599s) [upload=442241]: Starting upload Dec 26 15:26:13.096201371 (503h57m12.967284426s) [upload=442241]: Done starting upload Dec 26 15:26:13.096285331 (503h57m12.967368366s) [upload=442241]: Uploading segments... Dec 26 15:26:13.096634736 (503h57m12.967717781s) [upload=442241]: Got next segment. Inline: false Dec 26 15:26:13.096720009 (503h57m12.967803054s) [upload=442241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:26:13.096731480 (503h57m12.967814555s) [upload=442241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:26:13.346386396 (503h57m13.217469431s) [seg_pos={0 0}, upload=442241]: Begin upload segment... Dec 26 15:26:13.346431882 (503h57m13.217514917s) [seg_pos={0 0}, upload=442241]: Done begin upload segment. Dec 26 15:26:13.346455677 (503h57m13.217538723s) [noise=true , address=145.79.227.4:4186 , piece_id=TQBUGQ6SPETEJTLK4AK63X5F5SSZIUU6QKQDVQFV4H25Z2W2V54Q, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.346518798 (503h57m13.217601843s) [noise=true , address=145.79.227.51:4410 , piece_id=CKMYU2WJBXZBNV36LNRODJDVRUWHKX4N7F6QONY3ORA7FOPJZVZQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347012878 (503h57m13.218095924s) [noise=true , address=45.91.116.14:4191 , piece_id=6SUIRACEDO357ODZ5ZYBX6HRXA6KH72M467CEZZTGAHMSXVKHAVQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347077742 (503h57m13.218160787s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347456193 (503h57m13.218539238s) [noise=true , address=45.91.116.5:4299 , piece_id=2GZLZBNFNIZ44QMW3PKNU65WWWUBX5ZSQFUKWGBCIAKDMIYCJS7A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.347756103 (503h57m13.218839148s) [noise=true , address=45.91.116.10:4083 , piece_id=5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348078628 (503h57m13.219161703s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348395862 (503h57m13.219478907s) [noise=true , address=145.79.227.15:4876 , piece_id=YOOH7ACHJ2VL3ZJIZY3DL3ZUL42RCEKVFQDFZ22BXDEGLZWBPC5A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.348733134 (503h57m13.219816179s) [noise=true , address=77.90.31.18:4808 , piece_id=WAUTGUPSCCXPIV4VENHXEOB57FWC7OQK3GN2UAHHQREX53RJXVAQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.349016083 (503h57m13.220099128s) [noise=true , address=45.91.116.6:4973 , piece_id=KF27ZZMNPB5PRCUMMWSCAVSYRLMNHSO7L2NHCRW3RGKZ7JOI4RFA, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.351559577 (503h57m13.222642612s) [noise=true , address=145.79.227.39:4015 , piece_id=BMXFV5TLEC6WJBJFWJBITMAUWVTC3ZU56U7SIYLUTL3GXIEEMCNQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.351893382 (503h57m13.222976417s) [noise=true , address=145.79.227.48:4301 , piece_id=GVZGXRQUAF3TCT5GVQUSI6VT2XX6G5O6PUC4J26REL76VU5ZLALQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.352776163 (503h57m13.223859208s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.353302125 (503h57m13.22438517s) [noise=true , address=77.90.31.15:4700 , piece_id=DV5PGV2FOWNSONKYW5TVO2GEAHRKRFPH4VTQF2FISNKOQ7JFNEKQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.354596349 (503h57m13.225679394s) [noise=true , address=45.91.116.4:4940 , piece_id=CFSW3OH4JGHUTOWS2SR5ZQ6B7TZ5WQT7DGGD3K2ZUMLHEEEDWKHQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:13.355634927 (503h57m13.226717972s) [noise=true , address=45.91.116.11:4118 , piece_id=V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ, seg_pos={0 0}, upload=442241]: Uploading piece... Dec 26 15:26:15.062260982 (503h57m14.933344028s) [upload=442241]: Next returned nil segment Dec 26 15:26:15.062318943 (503h57m14.933401988s) [upload=442241]: Waiting for error group managing segments... Dec 26 15:26:15.344789265 (503h57m15.2158723s) [noise=true , address=145.79.227.4:4186 , piece_id=TQBUGQ6SPETEJTLK4AK63X5F5SSZIUU6QKQDVQFV4H25Z2W2V54Q, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.348322673 (503h57m15.219405708s) [noise=true , address=145.79.227.15:4876 , piece_id=YOOH7ACHJ2VL3ZJIZY3DL3ZUL42RCEKVFQDFZ22BXDEGLZWBPC5A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.360828747 (503h57m15.231911782s) [noise=true , address=145.79.227.51:4410 , piece_id=CKMYU2WJBXZBNV36LNRODJDVRUWHKX4N7F6QONY3ORA7FOPJZVZQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.370693872 (503h57m15.241776907s) [noise=true , address=145.79.227.39:4015 , piece_id=BMXFV5TLEC6WJBJFWJBITMAUWVTC3ZU56U7SIYLUTL3GXIEEMCNQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.384026740 (503h57m15.255109775s) [noise=true , address=145.79.227.48:4301 , piece_id=GVZGXRQUAF3TCT5GVQUSI6VT2XX6G5O6PUC4J26REL76VU5ZLALQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.604946208 (503h57m15.476029243s) [noise=true , address=45.91.116.5:4299 , piece_id=2GZLZBNFNIZ44QMW3PKNU65WWWUBX5ZSQFUKWGBCIAKDMIYCJS7A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.625370549 (503h57m15.496453584s) [noise=true , address=45.91.116.6:4973 , piece_id=KF27ZZMNPB5PRCUMMWSCAVSYRLMNHSO7L2NHCRW3RGKZ7JOI4RFA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.768372509 (503h57m15.639455544s) [noise=true , address=77.90.31.15:4700 , piece_id=DV5PGV2FOWNSONKYW5TVO2GEAHRKRFPH4VTQF2FISNKOQ7JFNEKQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.772904880 (503h57m15.643987925s) [noise=true , address=45.91.116.4:4940 , piece_id=CFSW3OH4JGHUTOWS2SR5ZQ6B7TZ5WQT7DGGD3K2ZUMLHEEEDWKHQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.776538659 (503h57m15.647621694s) [noise=true , address=45.91.116.11:4118 , piece_id=V4YHIK55FIY2OYNFTQMXQZNDJDGNJXM5FK7HFPXTTUX2CGTLWQOQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:15.825341898 (503h57m15.696424933s) [noise=true , address=45.91.116.14:4191 , piece_id=6SUIRACEDO357ODZ5ZYBX6HRXA6KH72M467CEZZTGAHMSXVKHAVQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.209263741 (503h57m16.080346786s) [noise=true , address=77.90.31.18:4808 , piece_id=WAUTGUPSCCXPIV4VENHXEOB57FWC7OQK3GN2UAHHQREX53RJXVAQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.481619854 (503h57m16.352702889s) [noise=true , address=45.91.116.10:4083 , piece_id=5PCWALUMBBBFA5XPE6SICK7ZJO2E7LYIF4BKHT3TAIZ2CCQDN5SA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: <nil> Dec 26 15:26:16.481638810 (503h57m16.352721845s) [seg_pos={0 0}, upload=442241]: Segment reached optimal threshold of 13 pieces. Dec 26 15:26:16.481856614 (503h57m16.352939649s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.481863618 (503h57m16.352946653s) [noise=true , address=145.79.227.33:4870 , piece_id=5DHMBJNFRDFA42XOC44EVS5NXQ6HEQZKMF5XR3B7HJSPJPISHXGA, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482081783 (503h57m16.353164819s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.482087865 (503h57m16.3531709s) [noise=true , address=45.91.116.9:4263 , piece_id=42L5YRJQDJEFET3JY6V2CDPWFHZBMIJCYR3MYLSY6CZJWX6QKPWQ, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482202613 (503h57m16.353285648s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 26 15:26:16.482208123 (503h57m16.353291169s) [noise=true , address=45.91.116.7:4012 , piece_id=ZCTRLS63ORIKO4NNK6ZL6P2QN64WTX6WTXBRTOBHZZRT7IJAJS6A, seg_pos={0 0}, upload=442241]: Long tail cancellation due to optimal threshold Dec 26 15:26:16.482224265 (503h57m16.3533073s) [seg_pos={0 0}, upload=442241]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:26:16.482285512 (503h57m16.353368547s) [upload=442241]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:26:16.482290240 (503h57m16.353373275s) [upload=442241]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:26:16.771425964 (503h57m16.642508999s) [upload=442241]: Done uploading segments... Dec 26 15:26:16.771446753 (503h57m16.642529798s) [upload=442241]: Upload finished. err: <nil> Duration: 30.00302846s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [..XX.X..XX..X.X..X..X.X...XX.XXXX.X..................X......] duration: 56.862867063s trace: https://jaeger.dev.storj.io/trace/649b436dfcc2315b FAIL (2025-12-26 15:26:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ploading piece... Dec 26 15:20:15.221423190 (503h51m15.092506225s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.221926739 (503h51m15.093009784s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222384111 (503h51m15.093467146s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222766368 (503h51m15.093849403s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222828496 (503h51m15.093911541s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223267943 (503h51m15.094350988s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223409813 (503h51m15.094492858s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223867144 (503h51m15.094950189s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224369451 (503h51m15.095452496s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224780333 (503h51m15.095863368s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225240039 (503h51m15.096323074s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225715373 (503h51m15.096798418s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.226705990 (503h51m15.097789025s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.227259905 (503h51m15.09834295s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:16.563815303 (503h51m16.434898348s) [upload=442149]: Next returned nil segment Dec 26 15:20:16.563835262 (503h51m16.434918337s) [upload=442149]: Waiting for error group managing segments... Dec 26 15:20:16.903963975 (503h51m16.77504702s) [noise=true , address=64.49.13.137:4500 , piece_id=E2KGA6VPKOQLQDONDE2T5PQVI4ENQ36C7BYS2MYWL2GDZ6KCZB5A, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.107984915 (503h51m16.97906795s) [noise=true , address=145.79.227.52:4466 , piece_id=7NVAHOJF72H3DBV5VZKGABRKIDIKTI6TIUCR6RVHHVZNSZA3PELQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.150952764 (503h51m17.022035799s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.179410862 (503h51m17.050493907s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.288743881 (503h51m17.159826927s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.315789470 (503h51m17.186872515s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.426193849 (503h51m17.297276894s) [noise=true , address=45.91.116.8:4074 , piece_id=4FHJLTE4GXQLPIZRZC2XZBM2F6JV35YJ7PEUI7SGAYG5PYZBBBCQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.433610592 (503h51m17.304693627s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.443740401 (503h51m17.314823446s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.990102417 (503h51m17.861185452s) [noise=true , address=145.79.227.44:4227 , piece_id=5QIQ2L6ZYNTQUFC44NHQ2MWLS6Q57UICGLGM2SSK5V7KZU5KR55Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072105685 (503h51m17.94318873s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072663447 (503h51m17.943746482s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072690278 (503h51m17.943773314s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133849817 (503h51m18.004932852s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133871317 (503h51m18.004954352s) [seg_pos={0 0}, upload=442149]: Segment reached optimal threshold of 14 pieces. Dec 26 15:20:18.134067501 (503h51m18.005150546s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 26 15:20:18.134078251 (503h51m18.005161296s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134103530 (503h51m18.005186575s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection Dec 26 15:20:18.134111224 (503h51m18.005194269s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134225823 (503h51m18.005308878s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection Dec 26 15:20:18.134234610 (503h51m18.005317645s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134587341 (503h51m18.005670396s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2): protocol: read tcp 188.166.225.170:57666->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:20:18.134596639 (503h51m18.005679684s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134618430 (503h51m18.005701465s) [seg_pos={0 0}, upload=442149]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:20:18.134714033 (503h51m18.005797078s) [upload=442149]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:20:18.134721266 (503h51m18.005804311s) [upload=442149]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:20:18.446337447 (503h51m18.317420492s) [upload=442149]: Done uploading segments... Dec 26 15:20:18.446364277 (503h51m18.317447322s) [upload=442149]: Upload finished. err: <nil> Duration: 56.862884827s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XXXXXXX..X..XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX..] duration: 5.001158126s trace: https://jaeger.dev.storj.io/trace/5c8ebbc50ab9d703 FAIL (2025-12-26 15:26:26 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001186721s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..XX...........XX..X.X.....X.....X.X............XXX....XX...] duration: 6.886445664s trace: https://jaeger.dev.storj.io/trace/25e9469c7d7223e7 FAIL (2025-12-26 15:25:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:23:38.415725880 (503h54m38.286808925s) [upload=442200]: Starting upload Dec 26 15:23:38.415825660 (503h54m38.286908715s) [upload=442200]: Done starting upload Dec 26 15:23:38.415905702 (503h54m38.286988737s) [upload=442200]: Uploading segments... Dec 26 15:23:38.416105463 (503h54m38.287188508s) [upload=442200]: Got next segment. Inline: false Dec 26 15:23:38.416144988 (503h54m38.287228033s) [upload=442200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:23:38.416151510 (503h54m38.287234555s) [upload=442200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:23:38.705120466 (503h54m38.576203521s) [seg_pos={0 0}, upload=442200]: Begin upload segment... Dec 26 15:23:38.705182634 (503h54m38.576265679s) [seg_pos={0 0}, upload=442200]: Done begin upload segment. Dec 26 15:23:38.705309696 (503h54m38.576392742s) [noise=true , address=145.79.227.37:4565 , piece_id=BC43N4MBN5O3NX4BSWOCOQE4WTPR6FPNZO6KGO7NM56HRXSD3L6Q, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.705343691 (503h54m38.576426736s) [noise=true , address=45.91.116.9:4263 , piece_id=XGHOYXSNNYWKNEFOOYEKPK7VQNQKMRZ6HD5B7PAG4YXUSS5J4G3A, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.705933695 (503h54m38.57701674s) [noise=true , address=145.79.227.13:4803 , piece_id=M5TUVGVYMFUKT2JJBB3GMISXB73L64EC7XQFND4HQ2NK7BSZPIFA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.706395113 (503h54m38.577478158s) [noise=true , address=145.79.227.48:4301 , piece_id=ER55FS4ASEV4546UDPVLZPRSBQJZNAGEFEA5GBKC44QA4IOXCNWA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.706884124 (503h54m38.577967179s) [noise=true , address=45.91.116.12:4225 , piece_id=BR6TXQRM5OWYJE7MLZN37MBBO4PVAMYJW23CFXZEDKVQKN2S2CTA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.707418552 (503h54m38.578501587s) [noise=true , address=145.79.227.3:4151 , piece_id=RIZQIUHBFIWEPMUZMOUYIZVESZDQ55TBK5MUHRCRTPVEVWQVFTDQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.707932200 (503h54m38.579015245s) [noise=true , address=145.79.227.11:4731 , piece_id=ZJE76EOILDIVHG53CWMEROKZTPBJTRAI6OIS6Y3MAVS7GONZEKKQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.709137225 (503h54m38.58022027s) [noise=true , address=145.79.227.31:4799 , piece_id=5SPUCFDQPRDTVKDFLEEQJZ4JTVCNGUYHVRY3BY275XAXDWHIR7PA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.710302864 (503h54m38.581385909s) [noise=true , address=45.91.116.10:4084 , piece_id=J2DPVXUXGOXCJB2KRJXTUFR3NELHXNPQN2DAQHZNCXJLLFFAVPLQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.710576225 (503h54m38.58165926s) [noise=true , address=145.79.227.6:4257 , piece_id=UP3MS7KZ3ZUAS2UWEISVLBUJXO5MIPIJIIMTCNXJ27PZCURCMU5Q, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.714177792 (503h54m38.585260897s) [noise=true , address=45.91.116.11:4118 , piece_id=4ST4YWZBXWINHTDOFHKWQYIPOAGUW3NA5SYBDMWDE6G57AXNSBTQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.722340756 (503h54m38.593423791s) [noise=true , address=145.79.227.15:4874 , piece_id=J7NEYAXDMWHGI6AL6J4RTVWQDPN34U3BRAWMAUFL6GXKCNVNB6LA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.723399081 (503h54m38.594482116s) [noise=true , address=145.79.227.47:4267 , piece_id=JXS66MLT5HALTBI7EURXVLNP6DFXCEJSLTCG7I4WPLN5Q5CLINVQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.724285399 (503h54m38.595368444s) [noise=true , address=45.91.116.13:4156 , piece_id=7DDWVUCITLY4I4RTLY4YGNXUZFTMNJ3BGHBSJQVW4WONJISVM3PA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.727316251 (503h54m38.598399296s) [noise=true , address=145.79.227.4:4185 , piece_id=HLGO7Q6SZ3ZEBEQRFCWV2NKUTQ72P7LKSYYE5AMFQ5CPQBDAIVQA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.727991977 (503h54m38.599075012s) [noise=true , address=45.91.116.14:4189 , piece_id=V3ZHIS76AWHW2XXSGRNXKB5ISUXP4TP4FF5K67VIUDZT5DFIY2WA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:40.437998390 (503h54m40.309081425s) [upload=442200]: Next returned nil segment Dec 26 15:23:40.438011956 (503h54m40.309095001s) [upload=442200]: Waiting for error group managing segments... Dec 26 15:23:40.721049898 (503h54m40.592132943s) [noise=true , address=145.79.227.37:4565 , piece_id=BC43N4MBN5O3NX4BSWOCOQE4WTPR6FPNZO6KGO7NM56HRXSD3L6Q, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.721954611 (503h54m40.593037656s) [noise=true , address=145.79.227.13:4803 , piece_id=M5TUVGVYMFUKT2JJBB3GMISXB73L64EC7XQFND4HQ2NK7BSZPIFA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.728413711 (503h54m40.599524218s) [noise=true , address=145.79.227.31:4799 , piece_id=5SPUCFDQPRDTVKDFLEEQJZ4JTVCNGUYHVRY3BY275XAXDWHIR7PA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.733969610 (503h54m40.605052655s) [noise=true , address=145.79.227.4:4185 , piece_id=HLGO7Q6SZ3ZEBEQRFCWV2NKUTQ72P7LKSYYE5AMFQ5CPQBDAIVQA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.741855877 (503h54m40.612938922s) [noise=true , address=145.79.227.11:4731 , piece_id=ZJE76EOILDIVHG53CWMEROKZTPBJTRAI6OIS6Y3MAVS7GONZEKKQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.745120773 (503h54m40.616203818s) [noise=true , address=145.79.227.48:4301 , piece_id=ER55FS4ASEV4546UDPVLZPRSBQJZNAGEFEA5GBKC44QA4IOXCNWA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.746226569 (503h54m40.617309614s) [noise=true , address=145.79.227.3:4151 , piece_id=RIZQIUHBFIWEPMUZMOUYIZVESZDQ55TBK5MUHRCRTPVEVWQVFTDQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.754619430 (503h54m40.625702475s) [noise=true , address=145.79.227.6:4257 , piece_id=UP3MS7KZ3ZUAS2UWEISVLBUJXO5MIPIJIIMTCNXJ27PZCURCMU5Q, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.755643571 (503h54m40.626726616s) [noise=true , address=145.79.227.47:4267 , piece_id=JXS66MLT5HALTBI7EURXVLNP6DFXCEJSLTCG7I4WPLN5Q5CLINVQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.758013403 (503h54m40.629096438s) [noise=true , address=145.79.227.15:4874 , piece_id=J7NEYAXDMWHGI6AL6J4RTVWQDPN34U3BRAWMAUFL6GXKCNVNB6LA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.957568680 (503h54m40.828651725s) [noise=true , address=45.91.116.10:4084 , piece_id=J2DPVXUXGOXCJB2KRJXTUFR3NELHXNPQN2DAQHZNCXJLLFFAVPLQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.966009763 (503h54m40.837092808s) [noise=true , address=45.91.116.9:4263 , piece_id=XGHOYXSNNYWKNEFOOYEKPK7VQNQKMRZ6HD5B7PAG4YXUSS5J4G3A, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:41.061908711 (503h54m40.932991746s) [noise=true , address=45.91.116.12:4225 , piece_id=BR6TXQRM5OWYJE7MLZN37MBBO4PVAMYJW23CFXZEDKVQKN2S2CTA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:41.061932537 (503h54m40.933015582s) [seg_pos={0 0}, upload=442200]: Segment reached optimal threshold of 13 pieces. Dec 26 15:23:41.062290759 (503h54m40.933373814s) [noise=true , address=45.91.116.14:4189 , piece_id=V3ZHIS76AWHW2XXSGRNXKB5ISUXP4TP4FF5K67VIUDZT5DFIY2WA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 15:23:41.062300278 (503h54m40.933383323s) [noise=true , address=45.91.116.14:4189 , piece_id=V3ZHIS76AWHW2XXSGRNXKB5ISUXP4TP4FF5K67VIUDZT5DFIY2WA, seg_pos={0 0}, upload=442200]: Long tail cancellation due to optimal threshold Dec 26 15:23:41.062752128 (503h54m40.933835193s) [noise=true , address=45.91.116.13:4156 , piece_id=7DDWVUCITLY4I4RTLY4YGNXUZFTMNJ3BGHBSJQVW4WONJISVM3PA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 15:23:41.062763409 (503h54m40.933846464s) [noise=true , address=45.91.116.13:4156 , piece_id=7DDWVUCITLY4I4RTLY4YGNXUZFTMNJ3BGHBSJQVW4WONJISVM3PA, seg_pos={0 0}, upload=442200]: Long tail cancellation due to optimal threshold Dec 26 15:23:41.062931389 (503h54m40.934014434s) [noise=true , address=45.91.116.11:4118 , piece_id=4ST4YWZBXWINHTDOFHKWQYIPOAGUW3NA5SYBDMWDE6G57AXNSBTQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 15:23:41.062940246 (503h54m40.934023291s) [noise=true , address=45.91.116.11:4118 , piece_id=4ST4YWZBXWINHTDOFHKWQYIPOAGUW3NA5SYBDMWDE6G57AXNSBTQ, seg_pos={0 0}, upload=442200]: Long tail cancellation due to optimal threshold Dec 26 15:23:41.062959523 (503h54m40.934042568s) [seg_pos={0 0}, upload=442200]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:23:41.063048793 (503h54m40.934131838s) [upload=442200]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:23:41.063054984 (503h54m40.934138029s) [upload=442200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:23:41.359680580 (503h54m41.230763615s) [upload=442200]: Done uploading segments... Dec 26 15:23:41.359704716 (503h54m41.230787751s) [upload=442200]: Upload finished. err: <nil> Duration: 6.886472175s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 18 out of 60 is failed (threshold=3) [...XX.X..XX..X.X..X..X.X...XX.XXXX.X..................X.....] duration: 49.178968531s trace: https://jaeger.dev.storj.io/trace/649b436dfcc2315b FAIL (2025-12-26 15:25:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ploading piece... Dec 26 15:20:15.221423190 (503h51m15.092506225s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.221926739 (503h51m15.093009784s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222384111 (503h51m15.093467146s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222766368 (503h51m15.093849403s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222828496 (503h51m15.093911541s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223267943 (503h51m15.094350988s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223409813 (503h51m15.094492858s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223867144 (503h51m15.094950189s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224369451 (503h51m15.095452496s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224780333 (503h51m15.095863368s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225240039 (503h51m15.096323074s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225715373 (503h51m15.096798418s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.226705990 (503h51m15.097789025s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.227259905 (503h51m15.09834295s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:16.563815303 (503h51m16.434898348s) [upload=442149]: Next returned nil segment Dec 26 15:20:16.563835262 (503h51m16.434918337s) [upload=442149]: Waiting for error group managing segments... Dec 26 15:20:16.903963975 (503h51m16.77504702s) [noise=true , address=64.49.13.137:4500 , piece_id=E2KGA6VPKOQLQDONDE2T5PQVI4ENQ36C7BYS2MYWL2GDZ6KCZB5A, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.107984915 (503h51m16.97906795s) [noise=true , address=145.79.227.52:4466 , piece_id=7NVAHOJF72H3DBV5VZKGABRKIDIKTI6TIUCR6RVHHVZNSZA3PELQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.150952764 (503h51m17.022035799s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.179410862 (503h51m17.050493907s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.288743881 (503h51m17.159826927s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.315789470 (503h51m17.186872515s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.426193849 (503h51m17.297276894s) [noise=true , address=45.91.116.8:4074 , piece_id=4FHJLTE4GXQLPIZRZC2XZBM2F6JV35YJ7PEUI7SGAYG5PYZBBBCQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.433610592 (503h51m17.304693627s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.443740401 (503h51m17.314823446s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.990102417 (503h51m17.861185452s) [noise=true , address=145.79.227.44:4227 , piece_id=5QIQ2L6ZYNTQUFC44NHQ2MWLS6Q57UICGLGM2SSK5V7KZU5KR55Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072105685 (503h51m17.94318873s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072663447 (503h51m17.943746482s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072690278 (503h51m17.943773314s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133849817 (503h51m18.004932852s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133871317 (503h51m18.004954352s) [seg_pos={0 0}, upload=442149]: Segment reached optimal threshold of 14 pieces. Dec 26 15:20:18.134067501 (503h51m18.005150546s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 26 15:20:18.134078251 (503h51m18.005161296s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134103530 (503h51m18.005186575s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection Dec 26 15:20:18.134111224 (503h51m18.005194269s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134225823 (503h51m18.005308878s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection Dec 26 15:20:18.134234610 (503h51m18.005317645s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134587341 (503h51m18.005670396s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2): protocol: read tcp 188.166.225.170:57666->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:20:18.134596639 (503h51m18.005679684s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134618430 (503h51m18.005701465s) [seg_pos={0 0}, upload=442149]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:20:18.134714033 (503h51m18.005797078s) [upload=442149]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:20:18.134721266 (503h51m18.005804311s) [upload=442149]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:20:18.446337447 (503h51m18.317420492s) [upload=442149]: Done uploading segments... Dec 26 15:20:18.446364277 (503h51m18.317447322s) [upload=442149]: Upload finished. err: <nil> Duration: 49.178983048s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [XXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX.] duration: 3.611677308s trace: https://jaeger.dev.storj.io/trace/5c8ebbc50ab9d703 FAIL (2025-12-26 15:25:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.611706965s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...XX...........XX..X.X.....X.....X.X............XXX....XX..] duration: 5.775820595s trace: https://jaeger.dev.storj.io/trace/25e9469c7d7223e7 FAIL (2025-12-26 15:25:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:23:38.415725880 (503h54m38.286808925s) [upload=442200]: Starting upload Dec 26 15:23:38.415825660 (503h54m38.286908715s) [upload=442200]: Done starting upload Dec 26 15:23:38.415905702 (503h54m38.286988737s) [upload=442200]: Uploading segments... Dec 26 15:23:38.416105463 (503h54m38.287188508s) [upload=442200]: Got next segment. Inline: false Dec 26 15:23:38.416144988 (503h54m38.287228033s) [upload=442200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:23:38.416151510 (503h54m38.287234555s) [upload=442200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:23:38.705120466 (503h54m38.576203521s) [seg_pos={0 0}, upload=442200]: Begin upload segment... Dec 26 15:23:38.705182634 (503h54m38.576265679s) [seg_pos={0 0}, upload=442200]: Done begin upload segment. Dec 26 15:23:38.705309696 (503h54m38.576392742s) [noise=true , address=145.79.227.37:4565 , piece_id=BC43N4MBN5O3NX4BSWOCOQE4WTPR6FPNZO6KGO7NM56HRXSD3L6Q, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.705343691 (503h54m38.576426736s) [noise=true , address=45.91.116.9:4263 , piece_id=XGHOYXSNNYWKNEFOOYEKPK7VQNQKMRZ6HD5B7PAG4YXUSS5J4G3A, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.705933695 (503h54m38.57701674s) [noise=true , address=145.79.227.13:4803 , piece_id=M5TUVGVYMFUKT2JJBB3GMISXB73L64EC7XQFND4HQ2NK7BSZPIFA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.706395113 (503h54m38.577478158s) [noise=true , address=145.79.227.48:4301 , piece_id=ER55FS4ASEV4546UDPVLZPRSBQJZNAGEFEA5GBKC44QA4IOXCNWA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.706884124 (503h54m38.577967179s) [noise=true , address=45.91.116.12:4225 , piece_id=BR6TXQRM5OWYJE7MLZN37MBBO4PVAMYJW23CFXZEDKVQKN2S2CTA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.707418552 (503h54m38.578501587s) [noise=true , address=145.79.227.3:4151 , piece_id=RIZQIUHBFIWEPMUZMOUYIZVESZDQ55TBK5MUHRCRTPVEVWQVFTDQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.707932200 (503h54m38.579015245s) [noise=true , address=145.79.227.11:4731 , piece_id=ZJE76EOILDIVHG53CWMEROKZTPBJTRAI6OIS6Y3MAVS7GONZEKKQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.709137225 (503h54m38.58022027s) [noise=true , address=145.79.227.31:4799 , piece_id=5SPUCFDQPRDTVKDFLEEQJZ4JTVCNGUYHVRY3BY275XAXDWHIR7PA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.710302864 (503h54m38.581385909s) [noise=true , address=45.91.116.10:4084 , piece_id=J2DPVXUXGOXCJB2KRJXTUFR3NELHXNPQN2DAQHZNCXJLLFFAVPLQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.710576225 (503h54m38.58165926s) [noise=true , address=145.79.227.6:4257 , piece_id=UP3MS7KZ3ZUAS2UWEISVLBUJXO5MIPIJIIMTCNXJ27PZCURCMU5Q, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.714177792 (503h54m38.585260897s) [noise=true , address=45.91.116.11:4118 , piece_id=4ST4YWZBXWINHTDOFHKWQYIPOAGUW3NA5SYBDMWDE6G57AXNSBTQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.722340756 (503h54m38.593423791s) [noise=true , address=145.79.227.15:4874 , piece_id=J7NEYAXDMWHGI6AL6J4RTVWQDPN34U3BRAWMAUFL6GXKCNVNB6LA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.723399081 (503h54m38.594482116s) [noise=true , address=145.79.227.47:4267 , piece_id=JXS66MLT5HALTBI7EURXVLNP6DFXCEJSLTCG7I4WPLN5Q5CLINVQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.724285399 (503h54m38.595368444s) [noise=true , address=45.91.116.13:4156 , piece_id=7DDWVUCITLY4I4RTLY4YGNXUZFTMNJ3BGHBSJQVW4WONJISVM3PA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.727316251 (503h54m38.598399296s) [noise=true , address=145.79.227.4:4185 , piece_id=HLGO7Q6SZ3ZEBEQRFCWV2NKUTQ72P7LKSYYE5AMFQ5CPQBDAIVQA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.727991977 (503h54m38.599075012s) [noise=true , address=45.91.116.14:4189 , piece_id=V3ZHIS76AWHW2XXSGRNXKB5ISUXP4TP4FF5K67VIUDZT5DFIY2WA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:40.437998390 (503h54m40.309081425s) [upload=442200]: Next returned nil segment Dec 26 15:23:40.438011956 (503h54m40.309095001s) [upload=442200]: Waiting for error group managing segments... Dec 26 15:23:40.721049898 (503h54m40.592132943s) [noise=true , address=145.79.227.37:4565 , piece_id=BC43N4MBN5O3NX4BSWOCOQE4WTPR6FPNZO6KGO7NM56HRXSD3L6Q, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.721954611 (503h54m40.593037656s) [noise=true , address=145.79.227.13:4803 , piece_id=M5TUVGVYMFUKT2JJBB3GMISXB73L64EC7XQFND4HQ2NK7BSZPIFA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.728413711 (503h54m40.599524218s) [noise=true , address=145.79.227.31:4799 , piece_id=5SPUCFDQPRDTVKDFLEEQJZ4JTVCNGUYHVRY3BY275XAXDWHIR7PA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.733969610 (503h54m40.605052655s) [noise=true , address=145.79.227.4:4185 , piece_id=HLGO7Q6SZ3ZEBEQRFCWV2NKUTQ72P7LKSYYE5AMFQ5CPQBDAIVQA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.741855877 (503h54m40.612938922s) [noise=true , address=145.79.227.11:4731 , piece_id=ZJE76EOILDIVHG53CWMEROKZTPBJTRAI6OIS6Y3MAVS7GONZEKKQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.745120773 (503h54m40.616203818s) [noise=true , address=145.79.227.48:4301 , piece_id=ER55FS4ASEV4546UDPVLZPRSBQJZNAGEFEA5GBKC44QA4IOXCNWA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.746226569 (503h54m40.617309614s) [noise=true , address=145.79.227.3:4151 , piece_id=RIZQIUHBFIWEPMUZMOUYIZVESZDQ55TBK5MUHRCRTPVEVWQVFTDQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.754619430 (503h54m40.625702475s) [noise=true , address=145.79.227.6:4257 , piece_id=UP3MS7KZ3ZUAS2UWEISVLBUJXO5MIPIJIIMTCNXJ27PZCURCMU5Q, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.755643571 (503h54m40.626726616s) [noise=true , address=145.79.227.47:4267 , piece_id=JXS66MLT5HALTBI7EURXVLNP6DFXCEJSLTCG7I4WPLN5Q5CLINVQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.758013403 (503h54m40.629096438s) [noise=true , address=145.79.227.15:4874 , piece_id=J7NEYAXDMWHGI6AL6J4RTVWQDPN34U3BRAWMAUFL6GXKCNVNB6LA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.957568680 (503h54m40.828651725s) [noise=true , address=45.91.116.10:4084 , piece_id=J2DPVXUXGOXCJB2KRJXTUFR3NELHXNPQN2DAQHZNCXJLLFFAVPLQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.966009763 (503h54m40.837092808s) [noise=true , address=45.91.116.9:4263 , piece_id=XGHOYXSNNYWKNEFOOYEKPK7VQNQKMRZ6HD5B7PAG4YXUSS5J4G3A, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:41.061908711 (503h54m40.932991746s) [noise=true , address=45.91.116.12:4225 , piece_id=BR6TXQRM5OWYJE7MLZN37MBBO4PVAMYJW23CFXZEDKVQKN2S2CTA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:41.061932537 (503h54m40.933015582s) [seg_pos={0 0}, upload=442200]: Segment reached optimal threshold of 13 pieces. Dec 26 15:23:41.062290759 (503h54m40.933373814s) [noise=true , address=45.91.116.14:4189 , piece_id=V3ZHIS76AWHW2XXSGRNXKB5ISUXP4TP4FF5K67VIUDZT5DFIY2WA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 15:23:41.062300278 (503h54m40.933383323s) [noise=true , address=45.91.116.14:4189 , piece_id=V3ZHIS76AWHW2XXSGRNXKB5ISUXP4TP4FF5K67VIUDZT5DFIY2WA, seg_pos={0 0}, upload=442200]: Long tail cancellation due to optimal threshold Dec 26 15:23:41.062752128 (503h54m40.933835193s) [noise=true , address=45.91.116.13:4156 , piece_id=7DDWVUCITLY4I4RTLY4YGNXUZFTMNJ3BGHBSJQVW4WONJISVM3PA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 15:23:41.062763409 (503h54m40.933846464s) [noise=true , address=45.91.116.13:4156 , piece_id=7DDWVUCITLY4I4RTLY4YGNXUZFTMNJ3BGHBSJQVW4WONJISVM3PA, seg_pos={0 0}, upload=442200]: Long tail cancellation due to optimal threshold Dec 26 15:23:41.062931389 (503h54m40.934014434s) [noise=true , address=45.91.116.11:4118 , piece_id=4ST4YWZBXWINHTDOFHKWQYIPOAGUW3NA5SYBDMWDE6G57AXNSBTQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 15:23:41.062940246 (503h54m40.934023291s) [noise=true , address=45.91.116.11:4118 , piece_id=4ST4YWZBXWINHTDOFHKWQYIPOAGUW3NA5SYBDMWDE6G57AXNSBTQ, seg_pos={0 0}, upload=442200]: Long tail cancellation due to optimal threshold Dec 26 15:23:41.062959523 (503h54m40.934042568s) [seg_pos={0 0}, upload=442200]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:23:41.063048793 (503h54m40.934131838s) [upload=442200]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:23:41.063054984 (503h54m40.934138029s) [upload=442200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:23:41.359680580 (503h54m41.230763615s) [upload=442200]: Done uploading segments... Dec 26 15:23:41.359704716 (503h54m41.230787751s) [upload=442200]: Upload finished. err: <nil> Duration: 5.775849771s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 19 out of 60 is failed (threshold=3) [X...XX.X..XX..X.X..X..X.X...XX.XXXX.X..................X....] duration: 49.15302869s trace: https://jaeger.dev.storj.io/trace/649b436dfcc2315b FAIL (2025-12-26 15:24:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ploading piece... Dec 26 15:20:15.221423190 (503h51m15.092506225s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.221926739 (503h51m15.093009784s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222384111 (503h51m15.093467146s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222766368 (503h51m15.093849403s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222828496 (503h51m15.093911541s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223267943 (503h51m15.094350988s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223409813 (503h51m15.094492858s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223867144 (503h51m15.094950189s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224369451 (503h51m15.095452496s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224780333 (503h51m15.095863368s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225240039 (503h51m15.096323074s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225715373 (503h51m15.096798418s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.226705990 (503h51m15.097789025s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.227259905 (503h51m15.09834295s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:16.563815303 (503h51m16.434898348s) [upload=442149]: Next returned nil segment Dec 26 15:20:16.563835262 (503h51m16.434918337s) [upload=442149]: Waiting for error group managing segments... Dec 26 15:20:16.903963975 (503h51m16.77504702s) [noise=true , address=64.49.13.137:4500 , piece_id=E2KGA6VPKOQLQDONDE2T5PQVI4ENQ36C7BYS2MYWL2GDZ6KCZB5A, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.107984915 (503h51m16.97906795s) [noise=true , address=145.79.227.52:4466 , piece_id=7NVAHOJF72H3DBV5VZKGABRKIDIKTI6TIUCR6RVHHVZNSZA3PELQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.150952764 (503h51m17.022035799s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.179410862 (503h51m17.050493907s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.288743881 (503h51m17.159826927s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.315789470 (503h51m17.186872515s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.426193849 (503h51m17.297276894s) [noise=true , address=45.91.116.8:4074 , piece_id=4FHJLTE4GXQLPIZRZC2XZBM2F6JV35YJ7PEUI7SGAYG5PYZBBBCQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.433610592 (503h51m17.304693627s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.443740401 (503h51m17.314823446s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.990102417 (503h51m17.861185452s) [noise=true , address=145.79.227.44:4227 , piece_id=5QIQ2L6ZYNTQUFC44NHQ2MWLS6Q57UICGLGM2SSK5V7KZU5KR55Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072105685 (503h51m17.94318873s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072663447 (503h51m17.943746482s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072690278 (503h51m17.943773314s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133849817 (503h51m18.004932852s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133871317 (503h51m18.004954352s) [seg_pos={0 0}, upload=442149]: Segment reached optimal threshold of 14 pieces. Dec 26 15:20:18.134067501 (503h51m18.005150546s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 26 15:20:18.134078251 (503h51m18.005161296s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134103530 (503h51m18.005186575s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection Dec 26 15:20:18.134111224 (503h51m18.005194269s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134225823 (503h51m18.005308878s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection Dec 26 15:20:18.134234610 (503h51m18.005317645s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134587341 (503h51m18.005670396s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2): protocol: read tcp 188.166.225.170:57666->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:20:18.134596639 (503h51m18.005679684s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134618430 (503h51m18.005701465s) [seg_pos={0 0}, upload=442149]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:20:18.134714033 (503h51m18.005797078s) [upload=442149]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:20:18.134721266 (503h51m18.005804311s) [upload=442149]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:20:18.446337447 (503h51m18.317420492s) [upload=442149]: Done uploading segments... Dec 26 15:20:18.446364277 (503h51m18.317447322s) [upload=442149]: Upload finished. err: <nil> Duration: 49.153046023s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [XXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.XX] duration: 5.001870162s trace: https://jaeger.dev.storj.io/trace/5c8ebbc50ab9d703 FAIL (2025-12-26 15:24:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001894097s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X...XX...........XX..X.X.....X.....X.X............XXX....XX.] duration: 5.743454955s trace: https://jaeger.dev.storj.io/trace/25e9469c7d7223e7 FAIL (2025-12-26 15:24:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:23:38.415725880 (503h54m38.286808925s) [upload=442200]: Starting upload Dec 26 15:23:38.415825660 (503h54m38.286908715s) [upload=442200]: Done starting upload Dec 26 15:23:38.415905702 (503h54m38.286988737s) [upload=442200]: Uploading segments... Dec 26 15:23:38.416105463 (503h54m38.287188508s) [upload=442200]: Got next segment. Inline: false Dec 26 15:23:38.416144988 (503h54m38.287228033s) [upload=442200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:23:38.416151510 (503h54m38.287234555s) [upload=442200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:23:38.705120466 (503h54m38.576203521s) [seg_pos={0 0}, upload=442200]: Begin upload segment... Dec 26 15:23:38.705182634 (503h54m38.576265679s) [seg_pos={0 0}, upload=442200]: Done begin upload segment. Dec 26 15:23:38.705309696 (503h54m38.576392742s) [noise=true , address=145.79.227.37:4565 , piece_id=BC43N4MBN5O3NX4BSWOCOQE4WTPR6FPNZO6KGO7NM56HRXSD3L6Q, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.705343691 (503h54m38.576426736s) [noise=true , address=45.91.116.9:4263 , piece_id=XGHOYXSNNYWKNEFOOYEKPK7VQNQKMRZ6HD5B7PAG4YXUSS5J4G3A, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.705933695 (503h54m38.57701674s) [noise=true , address=145.79.227.13:4803 , piece_id=M5TUVGVYMFUKT2JJBB3GMISXB73L64EC7XQFND4HQ2NK7BSZPIFA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.706395113 (503h54m38.577478158s) [noise=true , address=145.79.227.48:4301 , piece_id=ER55FS4ASEV4546UDPVLZPRSBQJZNAGEFEA5GBKC44QA4IOXCNWA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.706884124 (503h54m38.577967179s) [noise=true , address=45.91.116.12:4225 , piece_id=BR6TXQRM5OWYJE7MLZN37MBBO4PVAMYJW23CFXZEDKVQKN2S2CTA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.707418552 (503h54m38.578501587s) [noise=true , address=145.79.227.3:4151 , piece_id=RIZQIUHBFIWEPMUZMOUYIZVESZDQ55TBK5MUHRCRTPVEVWQVFTDQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.707932200 (503h54m38.579015245s) [noise=true , address=145.79.227.11:4731 , piece_id=ZJE76EOILDIVHG53CWMEROKZTPBJTRAI6OIS6Y3MAVS7GONZEKKQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.709137225 (503h54m38.58022027s) [noise=true , address=145.79.227.31:4799 , piece_id=5SPUCFDQPRDTVKDFLEEQJZ4JTVCNGUYHVRY3BY275XAXDWHIR7PA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.710302864 (503h54m38.581385909s) [noise=true , address=45.91.116.10:4084 , piece_id=J2DPVXUXGOXCJB2KRJXTUFR3NELHXNPQN2DAQHZNCXJLLFFAVPLQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.710576225 (503h54m38.58165926s) [noise=true , address=145.79.227.6:4257 , piece_id=UP3MS7KZ3ZUAS2UWEISVLBUJXO5MIPIJIIMTCNXJ27PZCURCMU5Q, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.714177792 (503h54m38.585260897s) [noise=true , address=45.91.116.11:4118 , piece_id=4ST4YWZBXWINHTDOFHKWQYIPOAGUW3NA5SYBDMWDE6G57AXNSBTQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.722340756 (503h54m38.593423791s) [noise=true , address=145.79.227.15:4874 , piece_id=J7NEYAXDMWHGI6AL6J4RTVWQDPN34U3BRAWMAUFL6GXKCNVNB6LA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.723399081 (503h54m38.594482116s) [noise=true , address=145.79.227.47:4267 , piece_id=JXS66MLT5HALTBI7EURXVLNP6DFXCEJSLTCG7I4WPLN5Q5CLINVQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.724285399 (503h54m38.595368444s) [noise=true , address=45.91.116.13:4156 , piece_id=7DDWVUCITLY4I4RTLY4YGNXUZFTMNJ3BGHBSJQVW4WONJISVM3PA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.727316251 (503h54m38.598399296s) [noise=true , address=145.79.227.4:4185 , piece_id=HLGO7Q6SZ3ZEBEQRFCWV2NKUTQ72P7LKSYYE5AMFQ5CPQBDAIVQA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.727991977 (503h54m38.599075012s) [noise=true , address=45.91.116.14:4189 , piece_id=V3ZHIS76AWHW2XXSGRNXKB5ISUXP4TP4FF5K67VIUDZT5DFIY2WA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:40.437998390 (503h54m40.309081425s) [upload=442200]: Next returned nil segment Dec 26 15:23:40.438011956 (503h54m40.309095001s) [upload=442200]: Waiting for error group managing segments... Dec 26 15:23:40.721049898 (503h54m40.592132943s) [noise=true , address=145.79.227.37:4565 , piece_id=BC43N4MBN5O3NX4BSWOCOQE4WTPR6FPNZO6KGO7NM56HRXSD3L6Q, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.721954611 (503h54m40.593037656s) [noise=true , address=145.79.227.13:4803 , piece_id=M5TUVGVYMFUKT2JJBB3GMISXB73L64EC7XQFND4HQ2NK7BSZPIFA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.728413711 (503h54m40.599524218s) [noise=true , address=145.79.227.31:4799 , piece_id=5SPUCFDQPRDTVKDFLEEQJZ4JTVCNGUYHVRY3BY275XAXDWHIR7PA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.733969610 (503h54m40.605052655s) [noise=true , address=145.79.227.4:4185 , piece_id=HLGO7Q6SZ3ZEBEQRFCWV2NKUTQ72P7LKSYYE5AMFQ5CPQBDAIVQA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.741855877 (503h54m40.612938922s) [noise=true , address=145.79.227.11:4731 , piece_id=ZJE76EOILDIVHG53CWMEROKZTPBJTRAI6OIS6Y3MAVS7GONZEKKQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.745120773 (503h54m40.616203818s) [noise=true , address=145.79.227.48:4301 , piece_id=ER55FS4ASEV4546UDPVLZPRSBQJZNAGEFEA5GBKC44QA4IOXCNWA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.746226569 (503h54m40.617309614s) [noise=true , address=145.79.227.3:4151 , piece_id=RIZQIUHBFIWEPMUZMOUYIZVESZDQ55TBK5MUHRCRTPVEVWQVFTDQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.754619430 (503h54m40.625702475s) [noise=true , address=145.79.227.6:4257 , piece_id=UP3MS7KZ3ZUAS2UWEISVLBUJXO5MIPIJIIMTCNXJ27PZCURCMU5Q, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.755643571 (503h54m40.626726616s) [noise=true , address=145.79.227.47:4267 , piece_id=JXS66MLT5HALTBI7EURXVLNP6DFXCEJSLTCG7I4WPLN5Q5CLINVQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.758013403 (503h54m40.629096438s) [noise=true , address=145.79.227.15:4874 , piece_id=J7NEYAXDMWHGI6AL6J4RTVWQDPN34U3BRAWMAUFL6GXKCNVNB6LA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.957568680 (503h54m40.828651725s) [noise=true , address=45.91.116.10:4084 , piece_id=J2DPVXUXGOXCJB2KRJXTUFR3NELHXNPQN2DAQHZNCXJLLFFAVPLQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.966009763 (503h54m40.837092808s) [noise=true , address=45.91.116.9:4263 , piece_id=XGHOYXSNNYWKNEFOOYEKPK7VQNQKMRZ6HD5B7PAG4YXUSS5J4G3A, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:41.061908711 (503h54m40.932991746s) [noise=true , address=45.91.116.12:4225 , piece_id=BR6TXQRM5OWYJE7MLZN37MBBO4PVAMYJW23CFXZEDKVQKN2S2CTA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:41.061932537 (503h54m40.933015582s) [seg_pos={0 0}, upload=442200]: Segment reached optimal threshold of 13 pieces. Dec 26 15:23:41.062290759 (503h54m40.933373814s) [noise=true , address=45.91.116.14:4189 , piece_id=V3ZHIS76AWHW2XXSGRNXKB5ISUXP4TP4FF5K67VIUDZT5DFIY2WA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 15:23:41.062300278 (503h54m40.933383323s) [noise=true , address=45.91.116.14:4189 , piece_id=V3ZHIS76AWHW2XXSGRNXKB5ISUXP4TP4FF5K67VIUDZT5DFIY2WA, seg_pos={0 0}, upload=442200]: Long tail cancellation due to optimal threshold Dec 26 15:23:41.062752128 (503h54m40.933835193s) [noise=true , address=45.91.116.13:4156 , piece_id=7DDWVUCITLY4I4RTLY4YGNXUZFTMNJ3BGHBSJQVW4WONJISVM3PA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 15:23:41.062763409 (503h54m40.933846464s) [noise=true , address=45.91.116.13:4156 , piece_id=7DDWVUCITLY4I4RTLY4YGNXUZFTMNJ3BGHBSJQVW4WONJISVM3PA, seg_pos={0 0}, upload=442200]: Long tail cancellation due to optimal threshold Dec 26 15:23:41.062931389 (503h54m40.934014434s) [noise=true , address=45.91.116.11:4118 , piece_id=4ST4YWZBXWINHTDOFHKWQYIPOAGUW3NA5SYBDMWDE6G57AXNSBTQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 15:23:41.062940246 (503h54m40.934023291s) [noise=true , address=45.91.116.11:4118 , piece_id=4ST4YWZBXWINHTDOFHKWQYIPOAGUW3NA5SYBDMWDE6G57AXNSBTQ, seg_pos={0 0}, upload=442200]: Long tail cancellation due to optimal threshold Dec 26 15:23:41.062959523 (503h54m40.934042568s) [seg_pos={0 0}, upload=442200]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:23:41.063048793 (503h54m40.934131838s) [upload=442200]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:23:41.063054984 (503h54m40.934138029s) [upload=442200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:23:41.359680580 (503h54m41.230763615s) [upload=442200]: Done uploading segments... Dec 26 15:23:41.359704716 (503h54m41.230787751s) [upload=442200]: Upload finished. err: <nil> Duration: 5.743484962s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [XXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.X] duration: 5.001237126s trace: https://jaeger.dev.storj.io/trace/3495535a52087a1e FAIL (2025-12-26 15:24:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001261201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XX...XX...........XX..X.X.....X.....X.X............XXX....XX] duration: 30.003611832s trace: https://jaeger.dev.storj.io/trace/25e9469c7d7223e7 FAIL (2025-12-26 15:24:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 15:23:38.415725880 (503h54m38.286808925s) [upload=442200]: Starting upload Dec 26 15:23:38.415825660 (503h54m38.286908715s) [upload=442200]: Done starting upload Dec 26 15:23:38.415905702 (503h54m38.286988737s) [upload=442200]: Uploading segments... Dec 26 15:23:38.416105463 (503h54m38.287188508s) [upload=442200]: Got next segment. Inline: false Dec 26 15:23:38.416144988 (503h54m38.287228033s) [upload=442200]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:23:38.416151510 (503h54m38.287234555s) [upload=442200]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:23:38.705120466 (503h54m38.576203521s) [seg_pos={0 0}, upload=442200]: Begin upload segment... Dec 26 15:23:38.705182634 (503h54m38.576265679s) [seg_pos={0 0}, upload=442200]: Done begin upload segment. Dec 26 15:23:38.705309696 (503h54m38.576392742s) [noise=true , address=145.79.227.37:4565 , piece_id=BC43N4MBN5O3NX4BSWOCOQE4WTPR6FPNZO6KGO7NM56HRXSD3L6Q, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.705343691 (503h54m38.576426736s) [noise=true , address=45.91.116.9:4263 , piece_id=XGHOYXSNNYWKNEFOOYEKPK7VQNQKMRZ6HD5B7PAG4YXUSS5J4G3A, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.705933695 (503h54m38.57701674s) [noise=true , address=145.79.227.13:4803 , piece_id=M5TUVGVYMFUKT2JJBB3GMISXB73L64EC7XQFND4HQ2NK7BSZPIFA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.706395113 (503h54m38.577478158s) [noise=true , address=145.79.227.48:4301 , piece_id=ER55FS4ASEV4546UDPVLZPRSBQJZNAGEFEA5GBKC44QA4IOXCNWA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.706884124 (503h54m38.577967179s) [noise=true , address=45.91.116.12:4225 , piece_id=BR6TXQRM5OWYJE7MLZN37MBBO4PVAMYJW23CFXZEDKVQKN2S2CTA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.707418552 (503h54m38.578501587s) [noise=true , address=145.79.227.3:4151 , piece_id=RIZQIUHBFIWEPMUZMOUYIZVESZDQ55TBK5MUHRCRTPVEVWQVFTDQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.707932200 (503h54m38.579015245s) [noise=true , address=145.79.227.11:4731 , piece_id=ZJE76EOILDIVHG53CWMEROKZTPBJTRAI6OIS6Y3MAVS7GONZEKKQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.709137225 (503h54m38.58022027s) [noise=true , address=145.79.227.31:4799 , piece_id=5SPUCFDQPRDTVKDFLEEQJZ4JTVCNGUYHVRY3BY275XAXDWHIR7PA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.710302864 (503h54m38.581385909s) [noise=true , address=45.91.116.10:4084 , piece_id=J2DPVXUXGOXCJB2KRJXTUFR3NELHXNPQN2DAQHZNCXJLLFFAVPLQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.710576225 (503h54m38.58165926s) [noise=true , address=145.79.227.6:4257 , piece_id=UP3MS7KZ3ZUAS2UWEISVLBUJXO5MIPIJIIMTCNXJ27PZCURCMU5Q, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.714177792 (503h54m38.585260897s) [noise=true , address=45.91.116.11:4118 , piece_id=4ST4YWZBXWINHTDOFHKWQYIPOAGUW3NA5SYBDMWDE6G57AXNSBTQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.722340756 (503h54m38.593423791s) [noise=true , address=145.79.227.15:4874 , piece_id=J7NEYAXDMWHGI6AL6J4RTVWQDPN34U3BRAWMAUFL6GXKCNVNB6LA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.723399081 (503h54m38.594482116s) [noise=true , address=145.79.227.47:4267 , piece_id=JXS66MLT5HALTBI7EURXVLNP6DFXCEJSLTCG7I4WPLN5Q5CLINVQ, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.724285399 (503h54m38.595368444s) [noise=true , address=45.91.116.13:4156 , piece_id=7DDWVUCITLY4I4RTLY4YGNXUZFTMNJ3BGHBSJQVW4WONJISVM3PA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.727316251 (503h54m38.598399296s) [noise=true , address=145.79.227.4:4185 , piece_id=HLGO7Q6SZ3ZEBEQRFCWV2NKUTQ72P7LKSYYE5AMFQ5CPQBDAIVQA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:38.727991977 (503h54m38.599075012s) [noise=true , address=45.91.116.14:4189 , piece_id=V3ZHIS76AWHW2XXSGRNXKB5ISUXP4TP4FF5K67VIUDZT5DFIY2WA, seg_pos={0 0}, upload=442200]: Uploading piece... Dec 26 15:23:40.437998390 (503h54m40.309081425s) [upload=442200]: Next returned nil segment Dec 26 15:23:40.438011956 (503h54m40.309095001s) [upload=442200]: Waiting for error group managing segments... Dec 26 15:23:40.721049898 (503h54m40.592132943s) [noise=true , address=145.79.227.37:4565 , piece_id=BC43N4MBN5O3NX4BSWOCOQE4WTPR6FPNZO6KGO7NM56HRXSD3L6Q, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.721954611 (503h54m40.593037656s) [noise=true , address=145.79.227.13:4803 , piece_id=M5TUVGVYMFUKT2JJBB3GMISXB73L64EC7XQFND4HQ2NK7BSZPIFA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.728413711 (503h54m40.599524218s) [noise=true , address=145.79.227.31:4799 , piece_id=5SPUCFDQPRDTVKDFLEEQJZ4JTVCNGUYHVRY3BY275XAXDWHIR7PA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.733969610 (503h54m40.605052655s) [noise=true , address=145.79.227.4:4185 , piece_id=HLGO7Q6SZ3ZEBEQRFCWV2NKUTQ72P7LKSYYE5AMFQ5CPQBDAIVQA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.741855877 (503h54m40.612938922s) [noise=true , address=145.79.227.11:4731 , piece_id=ZJE76EOILDIVHG53CWMEROKZTPBJTRAI6OIS6Y3MAVS7GONZEKKQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.745120773 (503h54m40.616203818s) [noise=true , address=145.79.227.48:4301 , piece_id=ER55FS4ASEV4546UDPVLZPRSBQJZNAGEFEA5GBKC44QA4IOXCNWA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.746226569 (503h54m40.617309614s) [noise=true , address=145.79.227.3:4151 , piece_id=RIZQIUHBFIWEPMUZMOUYIZVESZDQ55TBK5MUHRCRTPVEVWQVFTDQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.754619430 (503h54m40.625702475s) [noise=true , address=145.79.227.6:4257 , piece_id=UP3MS7KZ3ZUAS2UWEISVLBUJXO5MIPIJIIMTCNXJ27PZCURCMU5Q, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.755643571 (503h54m40.626726616s) [noise=true , address=145.79.227.47:4267 , piece_id=JXS66MLT5HALTBI7EURXVLNP6DFXCEJSLTCG7I4WPLN5Q5CLINVQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.758013403 (503h54m40.629096438s) [noise=true , address=145.79.227.15:4874 , piece_id=J7NEYAXDMWHGI6AL6J4RTVWQDPN34U3BRAWMAUFL6GXKCNVNB6LA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.957568680 (503h54m40.828651725s) [noise=true , address=45.91.116.10:4084 , piece_id=J2DPVXUXGOXCJB2KRJXTUFR3NELHXNPQN2DAQHZNCXJLLFFAVPLQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:40.966009763 (503h54m40.837092808s) [noise=true , address=45.91.116.9:4263 , piece_id=XGHOYXSNNYWKNEFOOYEKPK7VQNQKMRZ6HD5B7PAG4YXUSS5J4G3A, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:41.061908711 (503h54m40.932991746s) [noise=true , address=45.91.116.12:4225 , piece_id=BR6TXQRM5OWYJE7MLZN37MBBO4PVAMYJW23CFXZEDKVQKN2S2CTA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: <nil> Dec 26 15:23:41.061932537 (503h54m40.933015582s) [seg_pos={0 0}, upload=442200]: Segment reached optimal threshold of 13 pieces. Dec 26 15:23:41.062290759 (503h54m40.933373814s) [noise=true , address=45.91.116.14:4189 , piece_id=V3ZHIS76AWHW2XXSGRNXKB5ISUXP4TP4FF5K67VIUDZT5DFIY2WA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 26 15:23:41.062300278 (503h54m40.933383323s) [noise=true , address=45.91.116.14:4189 , piece_id=V3ZHIS76AWHW2XXSGRNXKB5ISUXP4TP4FF5K67VIUDZT5DFIY2WA, seg_pos={0 0}, upload=442200]: Long tail cancellation due to optimal threshold Dec 26 15:23:41.062752128 (503h54m40.933835193s) [noise=true , address=45.91.116.13:4156 , piece_id=7DDWVUCITLY4I4RTLY4YGNXUZFTMNJ3BGHBSJQVW4WONJISVM3PA, seg_pos={0 0}, upload=442200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 15:23:41.062763409 (503h54m40.933846464s) [noise=true , address=45.91.116.13:4156 , piece_id=7DDWVUCITLY4I4RTLY4YGNXUZFTMNJ3BGHBSJQVW4WONJISVM3PA, seg_pos={0 0}, upload=442200]: Long tail cancellation due to optimal threshold Dec 26 15:23:41.062931389 (503h54m40.934014434s) [noise=true , address=45.91.116.11:4118 , piece_id=4ST4YWZBXWINHTDOFHKWQYIPOAGUW3NA5SYBDMWDE6G57AXNSBTQ, seg_pos={0 0}, upload=442200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF Dec 26 15:23:41.062940246 (503h54m40.934023291s) [noise=true , address=45.91.116.11:4118 , piece_id=4ST4YWZBXWINHTDOFHKWQYIPOAGUW3NA5SYBDMWDE6G57AXNSBTQ, seg_pos={0 0}, upload=442200]: Long tail cancellation due to optimal threshold Dec 26 15:23:41.062959523 (503h54m40.934042568s) [seg_pos={0 0}, upload=442200]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:23:41.063048793 (503h54m40.934131838s) [upload=442200]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:23:41.063054984 (503h54m40.934138029s) [upload=442200]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:23:41.359680580 (503h54m41.230763615s) [upload=442200]: Done uploading segments... Dec 26 15:23:41.359704716 (503h54m41.230787751s) [upload=442200]: Upload finished. err: <nil> Duration: 30.00368459s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X...XX.X..XX..X.X..X..X.X...XX.XXXX.X..................X...] duration: 50.107610196s trace: https://jaeger.dev.storj.io/trace/649b436dfcc2315b FAIL (2025-12-26 15:24:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ploading piece... Dec 26 15:20:15.221423190 (503h51m15.092506225s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.221926739 (503h51m15.093009784s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222384111 (503h51m15.093467146s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222766368 (503h51m15.093849403s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222828496 (503h51m15.093911541s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223267943 (503h51m15.094350988s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223409813 (503h51m15.094492858s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223867144 (503h51m15.094950189s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224369451 (503h51m15.095452496s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224780333 (503h51m15.095863368s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225240039 (503h51m15.096323074s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225715373 (503h51m15.096798418s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.226705990 (503h51m15.097789025s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.227259905 (503h51m15.09834295s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:16.563815303 (503h51m16.434898348s) [upload=442149]: Next returned nil segment Dec 26 15:20:16.563835262 (503h51m16.434918337s) [upload=442149]: Waiting for error group managing segments... Dec 26 15:20:16.903963975 (503h51m16.77504702s) [noise=true , address=64.49.13.137:4500 , piece_id=E2KGA6VPKOQLQDONDE2T5PQVI4ENQ36C7BYS2MYWL2GDZ6KCZB5A, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.107984915 (503h51m16.97906795s) [noise=true , address=145.79.227.52:4466 , piece_id=7NVAHOJF72H3DBV5VZKGABRKIDIKTI6TIUCR6RVHHVZNSZA3PELQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.150952764 (503h51m17.022035799s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.179410862 (503h51m17.050493907s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.288743881 (503h51m17.159826927s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.315789470 (503h51m17.186872515s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.426193849 (503h51m17.297276894s) [noise=true , address=45.91.116.8:4074 , piece_id=4FHJLTE4GXQLPIZRZC2XZBM2F6JV35YJ7PEUI7SGAYG5PYZBBBCQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.433610592 (503h51m17.304693627s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.443740401 (503h51m17.314823446s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.990102417 (503h51m17.861185452s) [noise=true , address=145.79.227.44:4227 , piece_id=5QIQ2L6ZYNTQUFC44NHQ2MWLS6Q57UICGLGM2SSK5V7KZU5KR55Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072105685 (503h51m17.94318873s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072663447 (503h51m17.943746482s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072690278 (503h51m17.943773314s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133849817 (503h51m18.004932852s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133871317 (503h51m18.004954352s) [seg_pos={0 0}, upload=442149]: Segment reached optimal threshold of 14 pieces. Dec 26 15:20:18.134067501 (503h51m18.005150546s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 26 15:20:18.134078251 (503h51m18.005161296s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134103530 (503h51m18.005186575s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection Dec 26 15:20:18.134111224 (503h51m18.005194269s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134225823 (503h51m18.005308878s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection Dec 26 15:20:18.134234610 (503h51m18.005317645s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134587341 (503h51m18.005670396s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2): protocol: read tcp 188.166.225.170:57666->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:20:18.134596639 (503h51m18.005679684s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134618430 (503h51m18.005701465s) [seg_pos={0 0}, upload=442149]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:20:18.134714033 (503h51m18.005797078s) [upload=442149]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:20:18.134721266 (503h51m18.005804311s) [upload=442149]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:20:18.446337447 (503h51m18.317420492s) [upload=442149]: Done uploading segments... Dec 26 15:20:18.446364277 (503h51m18.317447322s) [upload=442149]: Upload finished. err: <nil> Duration: 50.107637938s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [XXXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X.] duration: 3.420513081s trace: https://jaeger.dev.storj.io/trace/6a31eada1e96d6c1 FAIL (2025-12-26 15:23:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.420538959s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXX...XX...........XX..X.X.....X.....X.X............XXX....X] duration: 30.002225751s trace: https://jaeger.dev.storj.io/trace/6fd295d6cb5533cf FAIL (2025-12-26 15:23:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: DCL3UPKXWKXD7JDNG333XRX3TUCTVQL26GYWUN74MNTOJQU36YDQ) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: GSPTRFRUXV3HDWLBMJL5CAYU3IAIMI6OGNUBGLKM2RHOT3VPHBQQ) context deadline exceeded ======LOGS====== Dec 26 15:23:08.195798957 (503h54m8.066881992s) [upload=442192]: Starting upload Dec 26 15:23:08.195885822 (503h54m8.066968898s) [upload=442192]: Done starting upload Dec 26 15:23:08.196133955 (503h54m8.06721699s) [upload=442192]: Uploading segments... Dec 26 15:23:08.196831964 (503h54m8.067914999s) [upload=442192]: Got next segment. Inline: false Dec 26 15:23:08.196887359 (503h54m8.067970404s) [upload=442192]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:23:08.196894413 (503h54m8.067977468s) [upload=442192]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:23:08.445793599 (503h54m8.316876644s) [seg_pos={0 0}, upload=442192]: Begin upload segment... Dec 26 15:23:08.446087519 (503h54m8.317170564s) [seg_pos={0 0}, upload=442192]: Done begin upload segment. Dec 26 15:23:08.446173482 (503h54m8.317256527s) [noise=true , address=145.79.227.51:4411 , piece_id=OSJY3HWUWJ4VWBLUILLSJKM3WIHAI2IYXRDZXSSHQFZ4DUTMBRXA, seg_pos={0 0}, upload=442192]: Uploading piece... Dec 26 15:23:08.446203810 (503h54m8.317286855s) [noise=true , address=145.79.227.50:4373 , piece_id=MPZK5VEZBLJA34EEJZIVU52U6TEU7N3FYS5K6ALTIWVO7PRR3IOA, seg_pos={0 0}, upload=442192]: Uploading piece... Dec 26 15:23:08.447439102 (503h54m8.318522147s) [noise=true , address=145.79.227.31:4799 , piece_id=3I4KGHKEAIRFGLN4M2HOWKGLX2LAGLBLILVWBRCYZKAXKB44SHBQ, seg_pos={0 0}, upload=442192]: Uploading piece... Dec 26 15:23:08.448273841 (503h54m8.319356887s) [noise=true , address=145.79.227.45:4230 , piece_id=ZPQ4FGGOYCHLBZPRTFPR5I2XHLWUWVMDFP5CQV4WREQFKG6AWRGA, seg_pos={0 0}, upload=442192]: Uploading piece... Dec 26 15:23:08.448930372 (503h54m8.320013427s) [noise=true , address=145.79.227.49:4338 , piece_id=6HIAOQOX6PWQ25JSPDVQUNX72MXVWDWXFK33EG4VCGBUETRSQPYQ, seg_pos={0 0}, upload=442192]: Uploading piece... Dec 26 15:23:08.450272658 (503h54m8.321355723s) [noise=true , address=145.79.227.2:4114 , piece_id=2UIGG627DBHMRLL3W5W4VBO5Z672LEUXWMJ4HPGV3SDRJXM3Z6VQ, seg_pos={0 0}, upload=442192]: Uploading piece... Dec 26 15:23:08.450450787 (503h54m8.321533832s) [noise=true , address=45.91.116.8:4047 , piece_id=DCL3UPKXWKXD7JDNG333XRX3TUCTVQL26GYWUN74MNTOJQU36YDQ, seg_pos={0 0}, upload=442192]: Uploading piece... Dec 26 15:23:08.451568936 (503h54m8.322651971s) [noise=true , address=45.91.116.13:4154 , piece_id=GSPTRFRUXV3HDWLBMJL5CAYU3IAIMI6OGNUBGLKM2RHOT3VPHBQQ, seg_pos={0 0}, upload=442192]: Uploading piece... Dec 26 15:23:08.451838871 (503h54m8.322921916s) [noise=true , address=145.79.227.15:4875 , piece_id=7XT5CTUMPUZG7IHKFDVYEHNXABKSEW7CKVSY2G2HHQRW5TFYR7TQ, seg_pos={0 0}, upload=442192]: Uploading piece... Dec 26 15:23:08.453419300 (503h54m8.324502335s) [noise=true , address=45.91.116.9:4261 , piece_id=E2R55NCDUWIS4TWC5XKQKWPSB7MSDBEDBNJSC3QE56C4ZNPQ3XBA, seg_pos={0 0}, upload=442192]: Uploading piece... Dec 26 15:23:08.454069146 (503h54m8.325152201s) [noise=true , address=45.91.116.11:4118 , piece_id=TZWRUFBWG6EM55SR3P4P5GSBQWOTR3OALAXCRHTD2UZGO76YXAIQ, seg_pos={0 0}, upload=442192]: Uploading piece... Dec 26 15:23:08.455522284 (503h54m8.326605339s) [noise=true , address=145.79.227.46:4602 , piece_id=HQTA7JVGKQ2OZOSPFO42CHETEX23OVCR3AT2YG2773VKM4TSKMHA, seg_pos={0 0}, upload=442192]: Uploading piece... Dec 26 15:23:08.457748743 (503h54m8.328831808s) [noise=true , address=145.79.227.5:4223 , piece_id=VCROL5BFGZT5YZJP2IO74NCGORMKPIMUN2Q25AREST3BIM7M7AFQ, seg_pos={0 0}, upload=442192]: Uploading piece... Dec 26 15:23:08.459693406 (503h54m8.330776451s) [noise=true , address=77.90.31.13:4627 , piece_id=TKGUFQMJZPMQLOEMYPCOSRAPCIWIDIMPXGLQDSANCBB44X7HKNHA, seg_pos={0 0}, upload=442192]: Uploading piece... Dec 26 15:23:08.461234330 (503h54m8.332317365s) [noise=true , address=145.79.227.47:4265 , piece_id=XUXRNUFNKBSRBA2A4SSGRP3D6AOCPX4RRJCEPP2DKBE7NLXRNFIA, seg_pos={0 0}, upload=442192]: Uploading piece... Dec 26 15:23:08.463667323 (503h54m8.334750368s) [noise=true , address=145.79.227.34:4907 , piece_id=2SPZL6HARF5ANBPTBB5JKSMMEXOVF5ES7RRVJ3TO4DRWEOPZNFWA, seg_pos={0 0}, upload=442192]: Uploading piece... Dec 26 15:23:10.166226135 (503h54m10.03730918s) [upload=442192]: Next returned nil segment Dec 26 15:23:10.166240934 (503h54m10.037323979s) [upload=442192]: Waiting for error group managing segments... Dec 26 15:23:10.451500636 (503h54m10.322583681s) [noise=true , address=145.79.227.2:4114 , piece_id=2UIGG627DBHMRLL3W5W4VBO5Z672LEUXWMJ4HPGV3SDRJXM3Z6VQ, seg_pos={0 0}, upload=442192]: Done uploading piece. err: <nil> Dec 26 15:23:10.453109249 (503h54m10.324192284s) [noise=true , address=145.79.227.49:4338 , piece_id=6HIAOQOX6PWQ25JSPDVQUNX72MXVWDWXFK33EG4VCGBUETRSQPYQ, seg_pos={0 0}, upload=442192]: Done uploading piece. err: <nil> Dec 26 15:23:10.453713049 (503h54m10.324796084s) [noise=true , address=145.79.227.46:4602 , piece_id=HQTA7JVGKQ2OZOSPFO42CHETEX23OVCR3AT2YG2773VKM4TSKMHA, seg_pos={0 0}, upload=442192]: Done uploading piece. err: <nil> Dec 26 15:23:10.461834784 (503h54m10.332917829s) [noise=true , address=145.79.227.15:4875 , piece_id=7XT5CTUMPUZG7IHKFDVYEHNXABKSEW7CKVSY2G2HHQRW5TFYR7TQ, seg_pos={0 0}, upload=442192]: Done uploading piece. err: <nil> Dec 26 15:23:10.463357033 (503h54m10.334440078s) [noise=true , address=145.79.227.31:4799 , piece_id=3I4KGHKEAIRFGLN4M2HOWKGLX2LAGLBLILVWBRCYZKAXKB44SHBQ, seg_pos={0 0}, upload=442192]: Done uploading piece. err: <nil> Dec 26 15:23:10.468635735 (503h54m10.33971878s) [noise=true , address=145.79.227.51:4411 , piece_id=OSJY3HWUWJ4VWBLUILLSJKM3WIHAI2IYXRDZXSSHQFZ4DUTMBRXA, seg_pos={0 0}, upload=442192]: Done uploading piece. err: <nil> Dec 26 15:23:10.471449863 (503h54m10.342532908s) [noise=true , address=145.79.227.45:4230 , piece_id=ZPQ4FGGOYCHLBZPRTFPR5I2XHLWUWVMDFP5CQV4WREQFKG6AWRGA, seg_pos={0 0}, upload=442192]: Done uploading piece. err: <nil> Dec 26 15:23:10.492160620 (503h54m10.363243665s) [noise=true , address=145.79.227.5:4223 , piece_id=VCROL5BFGZT5YZJP2IO74NCGORMKPIMUN2Q25AREST3BIM7M7AFQ, seg_pos={0 0}, upload=442192]: Done uploading piece. err: <nil> Dec 26 15:23:10.492550743 (503h54m10.363633778s) [noise=true , address=145.79.227.47:4265 , piece_id=XUXRNUFNKBSRBA2A4SSGRP3D6AOCPX4RRJCEPP2DKBE7NLXRNFIA, seg_pos={0 0}, upload=442192]: Done uploading piece. err: <nil> Dec 26 15:23:10.499731677 (503h54m10.370814712s) [noise=true , address=145.79.227.50:4373 , piece_id=MPZK5VEZBLJA34EEJZIVU52U6TEU7N3FYS5K6ALTIWVO7PRR3IOA, seg_pos={0 0}, upload=442192]: Done uploading piece. err: <nil> Dec 26 15:23:10.746422748 (503h54m10.617505783s) [noise=true , address=45.91.116.8:4047 , piece_id=DCL3UPKXWKXD7JDNG333XRX3TUCTVQL26GYWUN74MNTOJQU36YDQ, seg_pos={0 0}, upload=442192]: Done uploading piece. err: <nil> Dec 26 15:23:10.759173959 (503h54m10.630257004s) [noise=true , address=45.91.116.13:4154 , piece_id=GSPTRFRUXV3HDWLBMJL5CAYU3IAIMI6OGNUBGLKM2RHOT3VPHBQQ, seg_pos={0 0}, upload=442192]: Done uploading piece. err: <nil> Dec 26 15:23:10.819525188 (503h54m10.690608233s) [noise=true , address=45.91.116.11:4118 , piece_id=TZWRUFBWG6EM55SR3P4P5GSBQWOTR3OALAXCRHTD2UZGO76YXAIQ, seg_pos={0 0}, upload=442192]: Done uploading piece. err: <nil> Dec 26 15:23:10.819547570 (503h54m10.690630615s) [seg_pos={0 0}, upload=442192]: Segment reached optimal threshold of 13 pieces. Dec 26 15:23:10.819749134 (503h54m10.690832169s) [noise=true , address=45.91.116.9:4261 , piece_id=E2R55NCDUWIS4TWC5XKQKWPSB7MSDBEDBNJSC3QE56C4ZNPQ3XBA, seg_pos={0 0}, upload=442192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 26 15:23:10.819757450 (503h54m10.690840495s) [noise=true , address=45.91.116.9:4261 , piece_id=E2R55NCDUWIS4TWC5XKQKWPSB7MSDBEDBNJSC3QE56C4ZNPQ3XBA, seg_pos={0 0}, upload=442192]: Long tail cancellation due to optimal threshold Dec 26 15:23:10.819827152 (503h54m10.690910187s) [noise=true , address=145.79.227.34:4907 , piece_id=2SPZL6HARF5ANBPTBB5JKSMMEXOVF5ES7RRVJ3TO4DRWEOPZNFWA, seg_pos={0 0}, upload=442192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 15:23:10.819841751 (503h54m10.690924786s) [noise=true , address=145.79.227.34:4907 , piece_id=2SPZL6HARF5ANBPTBB5JKSMMEXOVF5ES7RRVJ3TO4DRWEOPZNFWA, seg_pos={0 0}, upload=442192]: Long tail cancellation due to optimal threshold Dec 26 15:23:10.819968061 (503h54m10.691051096s) [noise=true , address=77.90.31.13:4627 , piece_id=TKGUFQMJZPMQLOEMYPCOSRAPCIWIDIMPXGLQDSANCBB44X7HKNHA, seg_pos={0 0}, upload=442192]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: expected piece hash; context canceled; EOF Dec 26 15:23:10.819974754 (503h54m10.691057789s) [noise=true , address=77.90.31.13:4627 , piece_id=TKGUFQMJZPMQLOEMYPCOSRAPCIWIDIMPXGLQDSANCBB44X7HKNHA, seg_pos={0 0}, upload=442192]: Long tail cancellation due to optimal threshold Dec 26 15:23:10.819993800 (503h54m10.691076845s) [seg_pos={0 0}, upload=442192]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:23:10.820064925 (503h54m10.69114796s) [upload=442192]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:23:10.820069744 (503h54m10.691152779s) [upload=442192]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:23:11.132072091 (503h54m11.003155126s) [upload=442192]: Done uploading segments... Dec 26 15:23:11.132096777 (503h54m11.003179812s) [upload=442192]: Upload finished. err: <nil> Duration: 30.002249887s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X...XX.X..XX..X.X..X..X.X...XX.XXXX.X..................X..] duration: 56.81406588s trace: https://jaeger.dev.storj.io/trace/649b436dfcc2315b FAIL (2025-12-26 15:23:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ploading piece... Dec 26 15:20:15.221423190 (503h51m15.092506225s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.221926739 (503h51m15.093009784s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222384111 (503h51m15.093467146s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222766368 (503h51m15.093849403s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222828496 (503h51m15.093911541s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223267943 (503h51m15.094350988s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223409813 (503h51m15.094492858s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223867144 (503h51m15.094950189s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224369451 (503h51m15.095452496s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224780333 (503h51m15.095863368s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225240039 (503h51m15.096323074s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225715373 (503h51m15.096798418s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.226705990 (503h51m15.097789025s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.227259905 (503h51m15.09834295s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:16.563815303 (503h51m16.434898348s) [upload=442149]: Next returned nil segment Dec 26 15:20:16.563835262 (503h51m16.434918337s) [upload=442149]: Waiting for error group managing segments... Dec 26 15:20:16.903963975 (503h51m16.77504702s) [noise=true , address=64.49.13.137:4500 , piece_id=E2KGA6VPKOQLQDONDE2T5PQVI4ENQ36C7BYS2MYWL2GDZ6KCZB5A, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.107984915 (503h51m16.97906795s) [noise=true , address=145.79.227.52:4466 , piece_id=7NVAHOJF72H3DBV5VZKGABRKIDIKTI6TIUCR6RVHHVZNSZA3PELQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.150952764 (503h51m17.022035799s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.179410862 (503h51m17.050493907s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.288743881 (503h51m17.159826927s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.315789470 (503h51m17.186872515s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.426193849 (503h51m17.297276894s) [noise=true , address=45.91.116.8:4074 , piece_id=4FHJLTE4GXQLPIZRZC2XZBM2F6JV35YJ7PEUI7SGAYG5PYZBBBCQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.433610592 (503h51m17.304693627s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.443740401 (503h51m17.314823446s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.990102417 (503h51m17.861185452s) [noise=true , address=145.79.227.44:4227 , piece_id=5QIQ2L6ZYNTQUFC44NHQ2MWLS6Q57UICGLGM2SSK5V7KZU5KR55Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072105685 (503h51m17.94318873s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072663447 (503h51m17.943746482s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072690278 (503h51m17.943773314s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133849817 (503h51m18.004932852s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133871317 (503h51m18.004954352s) [seg_pos={0 0}, upload=442149]: Segment reached optimal threshold of 14 pieces. Dec 26 15:20:18.134067501 (503h51m18.005150546s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 26 15:20:18.134078251 (503h51m18.005161296s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134103530 (503h51m18.005186575s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection Dec 26 15:20:18.134111224 (503h51m18.005194269s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134225823 (503h51m18.005308878s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection Dec 26 15:20:18.134234610 (503h51m18.005317645s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134587341 (503h51m18.005670396s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2): protocol: read tcp 188.166.225.170:57666->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:20:18.134596639 (503h51m18.005679684s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134618430 (503h51m18.005701465s) [seg_pos={0 0}, upload=442149]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:20:18.134714033 (503h51m18.005797078s) [upload=442149]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:20:18.134721266 (503h51m18.005804311s) [upload=442149]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:20:18.446337447 (503h51m18.317420492s) [upload=442149]: Done uploading segments... Dec 26 15:20:18.446364277 (503h51m18.317447322s) [upload=442149]: Upload finished. err: <nil> Duration: 56.814083273s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XXXXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..X] duration: 5.001873969s trace: https://jaeger.dev.storj.io/trace/6a31eada1e96d6c1 FAIL (2025-12-26 15:22:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001906641s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.XXX...XX...........XX..X.X.....X.....X.X............XXX....] duration: 5.805435686s trace: https://jaeger.dev.storj.io/trace/43d5e4843af8c347 FAIL (2025-12-26 15:22:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: 6V7KN4TVW3DAM3WKDOJ6BISNBJXIIHB2OUICSAS7TBWBDPGIS2VQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 2IFCRUB7TM3GHS7NP4TJ7ECPSAUIRW3CNN7MZSBOWI7ES3RPNQHA) context deadline exceeded ======LOGS====== (503h50m3.651484732s) [upload=442133]: Uploading segments... Dec 26 15:19:03.780755370 (503h50m3.651838415s) [upload=442133]: Got next segment. Inline: false Dec 26 15:19:03.780798151 (503h50m3.651881196s) [upload=442133]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 15:19:03.780815615 (503h50m3.65189866s) [upload=442133]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 15:19:04.026178036 (503h50m3.897261081s) [seg_pos={0 0}, upload=442133]: Begin upload segment... Dec 26 15:19:04.026243440 (503h50m3.897326475s) [seg_pos={0 0}, upload=442133]: Done begin upload segment. Dec 26 15:19:04.026290710 (503h50m3.897373765s) [noise=true , address=145.79.227.2:4114 , piece_id=EHSPZFKOTAPUXUIVFRFTEEDFTRI7ZWMD5BCVT7J26ICOJUFTKYFA, seg_pos={0 0}, upload=442133]: Uploading piece... Dec 26 15:19:04.026348100 (503h50m3.897431145s) [noise=true , address=145.79.227.45:4231 , piece_id=QPCZN2T263F52OK5H4PLQ7QCSEYOR7QTRLQQLN5AGD7KUAARVDVA, seg_pos={0 0}, upload=442133]: Uploading piece... Dec 26 15:19:04.026464651 (503h50m3.897547696s) [noise=true , address=145.79.227.17:4972 , piece_id=SACJERTYBZMSIESUIYDKLMW5T3BHWGR5F77XYUYRIGTXUAHGVOUA, seg_pos={0 0}, upload=442133]: Uploading piece... Dec 26 15:19:04.027081817 (503h50m3.898164862s) [noise=true , address=145.79.227.15:4875 , piece_id=Q3GKVKXZA2SCJELGSWIW2EL4RMNWF7QFNK2TKIGDKJ5R47MQ6XBQ, seg_pos={0 0}, upload=442133]: Uploading piece... Dec 26 15:19:04.028584498 (503h50m3.899667553s) [noise=true , address=145.79.227.49:4337 , piece_id=GOGJAXUM545UXTDNCN3GIIA2UBRJL2QBJFMRVHL7BSX6B342MT2A, seg_pos={0 0}, upload=442133]: Uploading piece... Dec 26 15:19:04.029091093 (503h50m3.900174138s) [noise=true , address=45.91.116.8:4045 , piece_id=2NVS4HNEG6MQTE2Y7N4PBKJY23H5DNLVY7RZI7OU3RSKKMP57CPQ, seg_pos={0 0}, upload=442133]: Uploading piece... Dec 26 15:19:04.029689943 (503h50m3.900772998s) [noise=true , address=145.79.227.41:4087 , piece_id=WFAZAJG5CSQZ4P2ZCKUBAQJWWIMJT6QWEFTVWASJ2MOQBMUDRJ7A, seg_pos={0 0}, upload=442133]: Uploading piece... Dec 26 15:19:04.030286639 (503h50m3.901369694s) [noise=true , address=145.79.227.51:4411 , piece_id=ME6OFUM5OYDGPGKSHQ3LOKG345JUVGNWGRCRD7SSPZU4S7M7PQXA, seg_pos={0 0}, upload=442133]: Uploading piece... Dec 26 15:19:04.030710346 (503h50m3.901793391s) [noise=true , address=45.91.116.12:4227 , piece_id=6V7KN4TVW3DAM3WKDOJ6BISNBJXIIHB2OUICSAS7TBWBDPGIS2VQ, seg_pos={0 0}, upload=442133]: Uploading piece... Dec 26 15:19:04.031206531 (503h50m3.902289576s) [noise=true , address=45.91.116.13:4155 , piece_id=6R34UQLYQNFAW7UHPVXWOC5ZFH6X4V5O2AWEGXIPCY6J7LX55KUQ, seg_pos={0 0}, upload=442133]: Uploading piece... Dec 26 15:19:04.031763031 (503h50m3.902846076s) [noise=true , address=77.90.31.18:4808 , piece_id=3HPUEKDCT3DMXC3UUQMQVJV2ZJVE7HMC5HLESWXGE6NTME6BZTPQ, seg_pos={0 0}, upload=442133]: Uploading piece... Dec 26 15:19:04.032262121 (503h50m3.903345166s) [noise=true , address=45.91.116.14:4189 , piece_id=2IFCRUB7TM3GHS7NP4TJ7ECPSAUIRW3CNN7MZSBOWI7ES3RPNQHA, seg_pos={0 0}, upload=442133]: Uploading piece... Dec 26 15:19:04.032797570 (503h50m3.903880625s) [noise=true , address=77.90.31.17:4773 , piece_id=7JQOLQ6AGK77X7FX25IYIFRR5T2OQG5NGJU66HVFYBDXCI5G6PHQ, seg_pos={0 0}, upload=442133]: Uploading piece... Dec 26 15:19:04.033479549 (503h50m3.904562594s) [noise=true , address=145.79.227.29:4941 , piece_id=4PUVQVWAH7CIRS4XLW5OHBSMGXXLUOEE53O7HIWYBCVZOXYBVCCQ, seg_pos={0 0}, upload=442133]: Uploading piece... Dec 26 15:19:04.035240593 (503h50m3.906323638s) [noise=true , address=77.90.31.13:4627 , piece_id=XPYJMNYUPDJBGZVADKATPEBSDBPHHX4ROOC6WGVKPVSURDSBTBEA, seg_pos={0 0}, upload=442133]: Uploading piece... Dec 26 15:19:04.036920572 (503h50m3.908003617s) [noise=true , address=145.79.227.39:4013 , piece_id=7LLF7A3OJAOVGWOE63J5ZPVY7IG67RYFO2P7L2TKUEUHAO2KYG3Q, seg_pos={0 0}, upload=442133]: Uploading piece... Dec 26 15:19:05.740233148 (503h50m5.611316203s) [upload=442133]: Next returned nil segment Dec 26 15:19:05.740272744 (503h50m5.611355799s) [upload=442133]: Waiting for error group managing segments... Dec 26 15:19:06.033630115 (503h50m5.90471315s) [noise=true , address=145.79.227.45:4231 , piece_id=QPCZN2T263F52OK5H4PLQ7QCSEYOR7QTRLQQLN5AGD7KUAARVDVA, seg_pos={0 0}, upload=442133]: Done uploading piece. err: <nil> Dec 26 15:19:06.034119979 (503h50m5.905203014s) [noise=true , address=145.79.227.15:4875 , piece_id=Q3GKVKXZA2SCJELGSWIW2EL4RMNWF7QFNK2TKIGDKJ5R47MQ6XBQ, seg_pos={0 0}, upload=442133]: Done uploading piece. err: <nil> Dec 26 15:19:06.035516979 (503h50m5.906600024s) [noise=true , address=145.79.227.17:4972 , piece_id=SACJERTYBZMSIESUIYDKLMW5T3BHWGR5F77XYUYRIGTXUAHGVOUA, seg_pos={0 0}, upload=442133]: Done uploading piece. err: <nil> Dec 26 15:19:06.039202106 (503h50m5.910285151s) [noise=true , address=145.79.227.29:4941 , piece_id=4PUVQVWAH7CIRS4XLW5OHBSMGXXLUOEE53O7HIWYBCVZOXYBVCCQ, seg_pos={0 0}, upload=442133]: Done uploading piece. err: <nil> Dec 26 15:19:06.043316561 (503h50m5.914399596s) [noise=true , address=145.79.227.41:4087 , piece_id=WFAZAJG5CSQZ4P2ZCKUBAQJWWIMJT6QWEFTVWASJ2MOQBMUDRJ7A, seg_pos={0 0}, upload=442133]: Done uploading piece. err: <nil> Dec 26 15:19:06.044388622 (503h50m5.915471667s) [noise=true , address=145.79.227.51:4411 , piece_id=ME6OFUM5OYDGPGKSHQ3LOKG345JUVGNWGRCRD7SSPZU4S7M7PQXA, seg_pos={0 0}, upload=442133]: Done uploading piece. err: <nil> Dec 26 15:19:06.055077466 (503h50m5.926160511s) [noise=true , address=145.79.227.49:4337 , piece_id=GOGJAXUM545UXTDNCN3GIIA2UBRJL2QBJFMRVHL7BSX6B342MT2A, seg_pos={0 0}, upload=442133]: Done uploading piece. err: <nil> Dec 26 15:19:06.061533609 (503h50m5.932616654s) [noise=true , address=145.79.227.39:4013 , piece_id=7LLF7A3OJAOVGWOE63J5ZPVY7IG67RYFO2P7L2TKUEUHAO2KYG3Q, seg_pos={0 0}, upload=442133]: Done uploading piece. err: <nil> Dec 26 15:19:06.168555206 (503h50m6.039638251s) [noise=true , address=145.79.227.2:4114 , piece_id=EHSPZFKOTAPUXUIVFRFTEEDFTRI7ZWMD5BCVT7J26ICOJUFTKYFA, seg_pos={0 0}, upload=442133]: Done uploading piece. err: <nil> Dec 26 15:19:06.282162113 (503h50m6.153245158s) [noise=true , address=77.90.31.13:4627 , piece_id=XPYJMNYUPDJBGZVADKATPEBSDBPHHX4ROOC6WGVKPVSURDSBTBEA, seg_pos={0 0}, upload=442133]: Done uploading piece. err: <nil> Dec 26 15:19:06.308623379 (503h50m6.179706424s) [noise=true , address=45.91.116.12:4227 , piece_id=6V7KN4TVW3DAM3WKDOJ6BISNBJXIIHB2OUICSAS7TBWBDPGIS2VQ, seg_pos={0 0}, upload=442133]: Done uploading piece. err: <nil> Dec 26 15:19:06.320639530 (503h50m6.191722565s) [noise=true , address=45.91.116.13:4155 , piece_id=6R34UQLYQNFAW7UHPVXWOC5ZFH6X4V5O2AWEGXIPCY6J7LX55KUQ, seg_pos={0 0}, upload=442133]: Done uploading piece. err: <nil> Dec 26 15:19:06.384140307 (503h50m6.255223362s) [noise=true , address=45.91.116.14:4189 , piece_id=2IFCRUB7TM3GHS7NP4TJ7ECPSAUIRW3CNN7MZSBOWI7ES3RPNQHA, seg_pos={0 0}, upload=442133]: Done uploading piece. err: <nil> Dec 26 15:19:06.384187948 (503h50m6.255271003s) [seg_pos={0 0}, upload=442133]: Segment reached optimal threshold of 13 pieces. Dec 26 15:19:06.384397667 (503h50m6.255480702s) [noise=true , address=45.91.116.8:4045 , piece_id=2NVS4HNEG6MQTE2Y7N4PBKJY23H5DNLVY7RZI7OU3RSKKMP57CPQ, seg_pos={0 0}, upload=442133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: write tcp 188.166.225.170:53580->45.91.116.8:4045: use of closed network connection; write tcp 188.166.225.170:53580->45.91.116.8:4045: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53580->45.91.116.8:4045: use of closed network connection Dec 26 15:19:06.384405762 (503h50m6.255488807s) [noise=true , address=45.91.116.8:4045 , piece_id=2NVS4HNEG6MQTE2Y7N4PBKJY23H5DNLVY7RZI7OU3RSKKMP57CPQ, seg_pos={0 0}, upload=442133]: Long tail cancellation due to optimal threshold Dec 26 15:19:06.384481647 (503h50m6.255564703s) [noise=true , address=77.90.31.17:4773 , piece_id=7JQOLQ6AGK77X7FX25IYIFRR5T2OQG5NGJU66HVFYBDXCI5G6PHQ, seg_pos={0 0}, upload=442133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 26 15:19:06.384492448 (503h50m6.255575493s) [noise=true , address=77.90.31.17:4773 , piece_id=7JQOLQ6AGK77X7FX25IYIFRR5T2OQG5NGJU66HVFYBDXCI5G6PHQ, seg_pos={0 0}, upload=442133]: Long tail cancellation due to optimal threshold Dec 26 15:19:06.384666449 (503h50m6.255749494s) [noise=true , address=77.90.31.18:4808 , piece_id=3HPUEKDCT3DMXC3UUQMQVJV2ZJVE7HMC5HLESWXGE6NTME6BZTPQ, seg_pos={0 0}, upload=442133]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:60750->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:60750->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60750->77.90.31.18:4808: use of closed network connection Dec 26 15:19:06.384674464 (503h50m6.255757499s) [noise=true , address=77.90.31.18:4808 , piece_id=3HPUEKDCT3DMXC3UUQMQVJV2ZJVE7HMC5HLESWXGE6NTME6BZTPQ, seg_pos={0 0}, upload=442133]: Long tail cancellation due to optimal threshold Dec 26 15:19:06.384737074 (503h50m6.255820119s) [seg_pos={0 0}, upload=442133]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 15:19:06.384860538 (503h50m6.255943583s) [upload=442133]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:19:06.384868915 (503h50m6.25595195s) [upload=442133]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:19:06.694824222 (503h50m6.565907267s) [upload=442133]: Done uploading segments... Dec 26 15:19:06.694852045 (503h50m6.56593509s) [upload=442133]: Upload finished. err: <nil> Duration: 5.805462619s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX..X..XXXXXXXX.X.XX.XXXX..X...X.X.X........XXX..] duration: 5.000154685s trace: https://jaeger.dev.storj.io/trace/77f989ebcdc8c90f FAIL (2025-12-26 15:22:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000179813s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 19 out of 60 is failed (threshold=3) [...X...XX.X..XX..X.X..X..X.X...XX.XXXX.X..................X.] duration: 56.518990433s trace: https://jaeger.dev.storj.io/trace/649b436dfcc2315b FAIL (2025-12-26 15:22:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ploading piece... Dec 26 15:20:15.221423190 (503h51m15.092506225s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.221926739 (503h51m15.093009784s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222384111 (503h51m15.093467146s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222766368 (503h51m15.093849403s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.222828496 (503h51m15.093911541s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223267943 (503h51m15.094350988s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223409813 (503h51m15.094492858s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.223867144 (503h51m15.094950189s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224369451 (503h51m15.095452496s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.224780333 (503h51m15.095863368s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225240039 (503h51m15.096323074s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.225715373 (503h51m15.096798418s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.226705990 (503h51m15.097789025s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:15.227259905 (503h51m15.09834295s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Uploading piece... Dec 26 15:20:16.563815303 (503h51m16.434898348s) [upload=442149]: Next returned nil segment Dec 26 15:20:16.563835262 (503h51m16.434918337s) [upload=442149]: Waiting for error group managing segments... Dec 26 15:20:16.903963975 (503h51m16.77504702s) [noise=true , address=64.49.13.137:4500 , piece_id=E2KGA6VPKOQLQDONDE2T5PQVI4ENQ36C7BYS2MYWL2GDZ6KCZB5A, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.107984915 (503h51m16.97906795s) [noise=true , address=145.79.227.52:4466 , piece_id=7NVAHOJF72H3DBV5VZKGABRKIDIKTI6TIUCR6RVHHVZNSZA3PELQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.150952764 (503h51m17.022035799s) [noise=true , address=145.79.227.20:4444 , piece_id=A5JAOLZRQQZAADPEUJKDHF2OK6RWJI5Y7OW4EAPRTNVZJDYK6ULA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.179410862 (503h51m17.050493907s) [noise=true , address=145.79.227.5:4245 , piece_id=N5GEKRDAJKOHQNY753QT73AYFHYWLJTOJB5YZTBHJXFJ3264G27Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.288743881 (503h51m17.159826927s) [noise=true , address=77.90.31.11:4573 , piece_id=UAWLQ3YU4L4STBZP6TIRYCDQDR2KAZ44Y53THZK5ANL5UJDCI6OA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.315789470 (503h51m17.186872515s) [noise=true , address=45.91.116.7:4022 , piece_id=X2YCDSURJHXJQ2EKL2OOHHENMGJ44S4QNNNJAMGIYRTJH42UNL5Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.426193849 (503h51m17.297276894s) [noise=true , address=45.91.116.8:4074 , piece_id=4FHJLTE4GXQLPIZRZC2XZBM2F6JV35YJ7PEUI7SGAYG5PYZBBBCQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.433610592 (503h51m17.304693627s) [noise=true , address=77.90.31.21:4894 , piece_id=LMK3MFIUIDNUWM5NO3VRAEFFKYMVRZX57KNVGJBE2UWJK3YUUWNQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.443740401 (503h51m17.314823446s) [noise=true , address=77.90.31.19:4872 , piece_id=Q7OIBUEYEX2DVNL5LLFF42C5IP2TWWCFXELLS2PA3TA6DRE7SFRQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:17.990102417 (503h51m17.861185452s) [noise=true , address=145.79.227.44:4227 , piece_id=5QIQ2L6ZYNTQUFC44NHQ2MWLS6Q57UICGLGM2SSK5V7KZU5KR55Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072105685 (503h51m17.94318873s) [noise=true , address=77.90.31.12:4616 , piece_id=ZJVQ3ONMVZHC62G4OZP64IFMHEA4RQDJUVZFAVE5CRBTG5U6RWJA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072663447 (503h51m17.943746482s) [noise=true , address=145.79.227.30:4778 , piece_id=6JWIHBI3R3WIMUNWDGO7N3GJ4DQ6FRS7HYMQ3WKSZDW4FHKO7D7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.072690278 (503h51m17.943773314s) [noise=true , address=193.149.164.143:4700 , piece_id=PQJJGGY5RE7UZ6JERN5ZXYDIJWBSDWDR77ZDUY5EEJGPBA6SOUFQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133849817 (503h51m18.004932852s) [noise=true , address=77.90.31.14:4687 , piece_id=ZAH4AKZQ4RAV25CSWTMHHZ3JUUND6JFOMNYLI2EASUTV5R77C5HA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: <nil> Dec 26 15:20:18.133871317 (503h51m18.004954352s) [seg_pos={0 0}, upload=442149]: Segment reached optimal threshold of 14 pieces. Dec 26 15:20:18.134067501 (503h51m18.005150546s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 26 15:20:18.134078251 (503h51m18.005161296s) [noise=true , address=77.90.31.13:4652 , piece_id=O6OYEJ4PUYGNWJEEDAZGFGDY5MKGLIBWLLEL2AP6RIMVYUSUNW6Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134103530 (503h51m18.005186575s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57636->64.49.13.169:4705: use of closed network connection Dec 26 15:20:18.134111224 (503h51m18.005194269s) [noise=true , address=64.49.13.169:4705 , piece_id=GMX3Y7P45G34YM7GJXDVC75EBM2EKI6UXNAAERX2TU75KCPTD6SA, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134225823 (503h51m18.005308878s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35160->77.90.31.15:4718: use of closed network connection Dec 26 15:20:18.134234610 (503h51m18.005317645s) [noise=true , address=77.90.31.15:4718 , piece_id=5C4SQUDDMGIOWBM5KZ72GCZMDNHQRGS5KYJNHYVMI6KOQXJMODEQ, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134587341 (503h51m18.005670396s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ERdBm3XPw4s7JHeMWsVniwcqnfMtP4ZX8sBVjri6dzZtq8AF2): protocol: read tcp 188.166.225.170:57666->77.90.31.16:4747: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 15:20:18.134596639 (503h51m18.005679684s) [noise=true , address=77.90.31.16:4747 , piece_id=VNHTRIQGC2OYN2TBAGGOHAZB2YEBTWVDBTKEATFIETXHKTCVQX7Q, seg_pos={0 0}, upload=442149]: Long tail cancellation due to optimal threshold Dec 26 15:20:18.134618430 (503h51m18.005701465s) [seg_pos={0 0}, upload=442149]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 26 15:20:18.134714033 (503h51m18.005797078s) [upload=442149]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 15:20:18.134721266 (503h51m18.005804311s) [upload=442149]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 15:20:18.446337447 (503h51m18.317420492s) [upload=442149]: Done uploading segments... Dec 26 15:20:18.446364277 (503h51m18.317447322s) [upload=442149]: Upload finished. err: <nil> Duration: 56.519004792s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)