Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 2.207559737s trace: https://jaeger.dev.storj.io/trace/178b4cba623ae119 FAIL (2025-12-09 08:09:39 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.207584756s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.969543579s trace: https://jaeger.dev.storj.io/trace/7014f415ea72bfa2 FAIL (2025-12-09 08:09:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:56:08.451608077 (88h27m8.322691112s) [upload=77198]: Starting upload Dec 9 07:56:08.451664594 (88h27m8.322747639s) [upload=77198]: Done starting upload Dec 9 07:56:08.451743716 (88h27m8.322826761s) [upload=77198]: Uploading segments... Dec 9 07:56:08.451810052 (88h27m8.322893127s) [upload=77198]: Got next segment. Inline: true Dec 9 07:56:08.451818759 (88h27m8.322901804s) [upload=77198]: Waiting for error group managing segments... Dec 9 07:56:08.451893511 (88h27m8.322976546s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:56:08.451899602 (88h27m8.322982647s) [upload=77198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:56:08.451906044 (88h27m8.322989089s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:57:04.013681009 (88h28m3.884764054s) [upload=77198]: Done uploading segments... Dec 9 07:57:04.013737777 (88h28m3.884820822s) [upload=77198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.969569969s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.X.........................................................] duration: 5.628239974s trace: https://jaeger.dev.storj.io/trace/34e772a771ff36b2 FAIL (2025-12-09 08:09:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :29:22.029442363 (88h0m21.900525398s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.029471198 (88h0m21.900554233s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.030555563 (88h0m21.901638608s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031156617 (88h0m21.902239662s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031594381 (88h0m21.902677416s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032034670 (88h0m21.903117715s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032167963 (88h0m21.903250998s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032800087 (88h0m21.903883132s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:23.836496806 (88h0m23.707579841s) [upload=76810]: Next returned nil segment Dec 9 07:29:23.836512486 (88h0m23.707595531s) [upload=76810]: Waiting for error group managing segments... Dec 9 07:29:24.124325638 (88h0m23.995408683s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.114522896 (88h0m24.985605941s) [noise=true , address=145.79.227.21:4475 , piece_id=ED4VS3YEIFPJGGKW36BJKVLT3EXGVOFG3SD23LEPPFIAOOKHUD4Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.377083973 (88h0m25.248167008s) [noise=true , address=145.79.227.41:4086 , piece_id=EZUBNMEUWTNJBXJBWJYXFJQEPFFVQOUHZAWVJW6NCWKGRKAS2PSA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.480466423 (88h0m25.351549468s) [noise=true , address=145.79.227.42:4121 , piece_id=7I2TXNXZM2KM5TEDDF7Q5DDUSBAPBE3WWFTTJFGNM5JVVCTQNU5Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:26.767037853 (88h0m26.638120908s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:49.689337786 (88h0m49.560420841s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection Dec 9 07:29:49.689500304 (88h0m49.56058335s) [noise=true , address=145.79.227.34:4907 , piece_id=UG6M3T655YGW4OGBYEVOLQVR7U5XWXPDDNXBZAPTHYP2DYWDRRYA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection Dec 9 07:29:49.689541303 (88h0m49.560624348s) [noise=true , address=145.79.227.33:4870 , piece_id=6Q5XSDCJCDRWHPO7JWCDNPQM5JN2ESDCRXWY5V23AGA6Q4C5Y7HQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection Dec 9 07:29:49.689654439 (88h0m49.560737474s) [noise=true , address=145.79.227.29:4943 , piece_id=FQ2FNNQR6QBWTJOIQBIOU7BSWIA6ZZMDWC4DE42NUNR5Z7ZDGCGQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection Dec 9 07:29:49.689700015 (88h0m49.56078306s) [noise=true , address=145.79.227.42:4122 , piece_id=5WKTAWBKWCZVU3YX4RSF4FQHQABNCZUHTXMTK3AIHUAGYDQXQYVQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection Dec 9 07:29:49.689861212 (88h0m49.560944247s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection Dec 9 07:29:49.689914644 (88h0m49.560997689s) [noise=true , address=145.79.227.28:4725 , piece_id=AGDXVUHDFM7DC2HNW3YUKREAROODLZBIVUMTFYBY3CXYQ7HSYGWA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection Dec 9 07:29:49.690003323 (88h0m49.561086368s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection Dec 9 07:29:49.690090949 (88h0m49.561173994s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection Dec 9 07:29:49.690262546 (88h0m49.561345591s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection Dec 9 07:29:49.690582726 (88h0m49.561665772s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:50466->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:29:49.690631769 (88h0m49.561714805s) [seg_pos={0 0}, upload=76810]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Dec 9 07:29:50.737500571 (88h0m50.608583616s) [upload=76810]: Done uploading segments... Dec 9 07:29:50.737550175 (88h0m50.60863321s) [upload=76810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.62827973s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X...................................................X.] duration: 5.862987596s trace: https://jaeger.dev.storj.io/trace/215347873232f4ea FAIL (2025-12-09 08:09:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.863002374s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [....X.XX.XX..X..................X...........................] duration: 3.514167158s trace: https://jaeger.dev.storj.io/trace/610b5afc07f67951 FAIL (2025-12-09 08:09:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.514194229s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X....................X..............................] duration: 5.772899969s trace: https://jaeger.dev.storj.io/trace/7fe88be41ea8781e FAIL (2025-12-09 08:09:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng piece... Dec 9 07:45:33.502533729 (88h16m33.373616774s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503057677 (88h16m33.374140712s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503561135 (88h16m33.37464417s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504037222 (88h16m33.375120257s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504531994 (88h16m33.375615039s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505014954 (88h16m33.376098029s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505576262 (88h16m33.376659307s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.506942875 (88h16m33.37802591s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508063960 (88h16m33.379147005s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508207313 (88h16m33.379290358s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.509568444 (88h16m33.38065149s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.511408599 (88h16m33.382491644s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.513502646 (88h16m33.384585681s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.515541418 (88h16m33.386624484s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.517269299 (88h16m33.388352334s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.518893461 (88h16m33.389976496s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:34.970483840 (88h16m34.841566885s) [upload=77044]: Next returned nil segment Dec 9 07:45:34.970497998 (88h16m34.841581043s) [upload=77044]: Waiting for error group managing segments... Dec 9 07:45:35.375513876 (88h16m35.246596921s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.378851702 (88h16m35.249934747s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.452093509 (88h16m35.323176544s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.484823964 (88h16m35.355906999s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.501356315 (88h16m35.37243935s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.506575192 (88h16m35.377658238s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.511021099 (88h16m35.382104134s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.519588893 (88h16m35.390671938s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.540564915 (88h16m35.41164795s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.555394233 (88h16m35.426477278s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.565337939 (88h16m35.436420984s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.576688943 (88h16m35.447771978s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.589009313 (88h16m35.460092368s) [noise=true , address=45.91.116.13:4181 , piece_id=X6IEEUKM5GDXZD2URW6YAPSN4DXWOCD2NK56K5PFFAYEHL2RTJUQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619301786 (88h16m35.490384821s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619323929 (88h16m35.490406964s) [seg_pos={0 0}, upload=77044]: Segment reached optimal threshold of 14 pieces. Dec 9 07:45:35.619554758 (88h16m35.490637793s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619564476 (88h16m35.490647531s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619577912 (88h16m35.490660947s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619583001 (88h16m35.490666037s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619751342 (88h16m35.490834377s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619757253 (88h16m35.490840298s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619853827 (88h16m35.490936862s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection Dec 9 07:45:35.619862484 (88h16m35.490945519s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619883484 (88h16m35.490966519s) [seg_pos={0 0}, upload=77044]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:45:35.619956242 (88h16m35.491039288s) [upload=77044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:45:35.619961382 (88h16m35.491044417s) [upload=77044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:47:56.102293256 (88h18m55.973376291s) [upload=77044]: Done uploading segments... Dec 9 07:47:56.102346187 (88h18m55.973429222s) [upload=77044]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.772924626s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X...................................................X] duration: 6m40.123136323s trace: https://jaeger.dev.storj.io/trace/215347873232f4ea FAIL (2025-12-09 08:09:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 6m40.123162714s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [....XX..X.......X...X..................X.............X......] duration: 5.746125694s trace: https://jaeger.dev.storj.io/trace/2e6e846a5a22deeb FAIL (2025-12-09 08:09:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.746154118s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.948243948s trace: https://jaeger.dev.storj.io/trace/55634d9c1d55ff67 FAIL (2025-12-09 08:09:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.948272524s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 4.817908376s trace: https://jaeger.dev.storj.io/trace/50f433cc1e60731c FAIL (2025-12-09 08:09:10 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.817941088s core uplink eu1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 461.370871ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt trace: https://jaeger.dev.storj.io/trace/51a280a8635ce0b9 FAIL (2025-12-09 08:09:04 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 461.504937ms edge linksharing us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 690.984601ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/43834d9fac36e837 FAIL (2025-12-09 08:09:01 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 691.02009ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.96719135s trace: https://jaeger.dev.storj.io/trace/42dcab7a5e240fd8 FAIL (2025-12-09 08:08:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:28:09.412200246 (87h59m9.283283281s) [upload=76792]: Starting upload Dec 9 07:28:09.412257335 (87h59m9.283340371s) [upload=76792]: Done starting upload Dec 9 07:28:09.412337338 (87h59m9.283420383s) [upload=76792]: Uploading segments... Dec 9 07:28:09.412384597 (87h59m9.283467632s) [upload=76792]: Got next segment. Inline: true Dec 9 07:28:09.412398334 (87h59m9.283481389s) [upload=76792]: Waiting for error group managing segments... Dec 9 07:28:09.412446225 (87h59m9.28352927s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:28:09.412452456 (87h59m9.283535491s) [upload=76792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:28:09.412455432 (87h59m9.283538477s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:28:19.867794892 (87h59m19.738877927s) [upload=76792]: Done uploading segments... Dec 9 07:28:19.867843895 (87h59m19.73892694s) [upload=76792]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.967214124s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.....XX..X.......X...X..................X.............X.....] duration: 5.675057709s trace: https://jaeger.dev.storj.io/trace/2e6e846a5a22deeb FAIL (2025-12-09 08:08:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.67507904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.98093395s trace: https://jaeger.dev.storj.io/trace/7014f415ea72bfa2 FAIL (2025-12-09 08:08:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:56:08.451608077 (88h27m8.322691112s) [upload=77198]: Starting upload Dec 9 07:56:08.451664594 (88h27m8.322747639s) [upload=77198]: Done starting upload Dec 9 07:56:08.451743716 (88h27m8.322826761s) [upload=77198]: Uploading segments... Dec 9 07:56:08.451810052 (88h27m8.322893127s) [upload=77198]: Got next segment. Inline: true Dec 9 07:56:08.451818759 (88h27m8.322901804s) [upload=77198]: Waiting for error group managing segments... Dec 9 07:56:08.451893511 (88h27m8.322976546s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:56:08.451899602 (88h27m8.322982647s) [upload=77198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:56:08.451906044 (88h27m8.322989089s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:57:04.013681009 (88h28m3.884764054s) [upload=77198]: Done uploading segments... Dec 9 07:57:04.013737777 (88h28m3.884820822s) [upload=77198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.980952747s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 5.630105208s trace: https://jaeger.dev.storj.io/trace/5f10a1154ce06783 FAIL (2025-12-09 08:08:54 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.630129274s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XX.X........................................................] duration: 6.393015069s trace: https://jaeger.dev.storj.io/trace/34e772a771ff36b2 FAIL (2025-12-09 08:08:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :29:22.029442363 (88h0m21.900525398s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.029471198 (88h0m21.900554233s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.030555563 (88h0m21.901638608s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031156617 (88h0m21.902239662s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031594381 (88h0m21.902677416s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032034670 (88h0m21.903117715s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032167963 (88h0m21.903250998s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032800087 (88h0m21.903883132s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:23.836496806 (88h0m23.707579841s) [upload=76810]: Next returned nil segment Dec 9 07:29:23.836512486 (88h0m23.707595531s) [upload=76810]: Waiting for error group managing segments... Dec 9 07:29:24.124325638 (88h0m23.995408683s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.114522896 (88h0m24.985605941s) [noise=true , address=145.79.227.21:4475 , piece_id=ED4VS3YEIFPJGGKW36BJKVLT3EXGVOFG3SD23LEPPFIAOOKHUD4Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.377083973 (88h0m25.248167008s) [noise=true , address=145.79.227.41:4086 , piece_id=EZUBNMEUWTNJBXJBWJYXFJQEPFFVQOUHZAWVJW6NCWKGRKAS2PSA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.480466423 (88h0m25.351549468s) [noise=true , address=145.79.227.42:4121 , piece_id=7I2TXNXZM2KM5TEDDF7Q5DDUSBAPBE3WWFTTJFGNM5JVVCTQNU5Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:26.767037853 (88h0m26.638120908s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:49.689337786 (88h0m49.560420841s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection Dec 9 07:29:49.689500304 (88h0m49.56058335s) [noise=true , address=145.79.227.34:4907 , piece_id=UG6M3T655YGW4OGBYEVOLQVR7U5XWXPDDNXBZAPTHYP2DYWDRRYA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection Dec 9 07:29:49.689541303 (88h0m49.560624348s) [noise=true , address=145.79.227.33:4870 , piece_id=6Q5XSDCJCDRWHPO7JWCDNPQM5JN2ESDCRXWY5V23AGA6Q4C5Y7HQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection Dec 9 07:29:49.689654439 (88h0m49.560737474s) [noise=true , address=145.79.227.29:4943 , piece_id=FQ2FNNQR6QBWTJOIQBIOU7BSWIA6ZZMDWC4DE42NUNR5Z7ZDGCGQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection Dec 9 07:29:49.689700015 (88h0m49.56078306s) [noise=true , address=145.79.227.42:4122 , piece_id=5WKTAWBKWCZVU3YX4RSF4FQHQABNCZUHTXMTK3AIHUAGYDQXQYVQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection Dec 9 07:29:49.689861212 (88h0m49.560944247s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection Dec 9 07:29:49.689914644 (88h0m49.560997689s) [noise=true , address=145.79.227.28:4725 , piece_id=AGDXVUHDFM7DC2HNW3YUKREAROODLZBIVUMTFYBY3CXYQ7HSYGWA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection Dec 9 07:29:49.690003323 (88h0m49.561086368s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection Dec 9 07:29:49.690090949 (88h0m49.561173994s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection Dec 9 07:29:49.690262546 (88h0m49.561345591s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection Dec 9 07:29:49.690582726 (88h0m49.561665772s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:50466->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:29:49.690631769 (88h0m49.561714805s) [seg_pos={0 0}, upload=76810]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Dec 9 07:29:50.737500571 (88h0m50.608583616s) [upload=76810]: Done uploading segments... Dec 9 07:29:50.737550175 (88h0m50.60863321s) [upload=76810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.393043092s core uplink us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 442.331045ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt trace: https://jaeger.dev.storj.io/trace/51a280a8635ce0b9 FAIL (2025-12-09 08:08:46 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 442.382313ms edge linksharing us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [......XX..X.......X...X..................X.............X....] duration: 6.347914159s trace: https://jaeger.dev.storj.io/trace/2e6e846a5a22deeb FAIL (2025-12-09 08:08:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.347939678s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X....................X.............................] duration: 6.281127647s trace: https://jaeger.dev.storj.io/trace/7fe88be41ea8781e FAIL (2025-12-09 08:08:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng piece... Dec 9 07:45:33.502533729 (88h16m33.373616774s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503057677 (88h16m33.374140712s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503561135 (88h16m33.37464417s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504037222 (88h16m33.375120257s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504531994 (88h16m33.375615039s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505014954 (88h16m33.376098029s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505576262 (88h16m33.376659307s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.506942875 (88h16m33.37802591s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508063960 (88h16m33.379147005s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508207313 (88h16m33.379290358s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.509568444 (88h16m33.38065149s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.511408599 (88h16m33.382491644s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.513502646 (88h16m33.384585681s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.515541418 (88h16m33.386624484s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.517269299 (88h16m33.388352334s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.518893461 (88h16m33.389976496s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:34.970483840 (88h16m34.841566885s) [upload=77044]: Next returned nil segment Dec 9 07:45:34.970497998 (88h16m34.841581043s) [upload=77044]: Waiting for error group managing segments... Dec 9 07:45:35.375513876 (88h16m35.246596921s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.378851702 (88h16m35.249934747s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.452093509 (88h16m35.323176544s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.484823964 (88h16m35.355906999s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.501356315 (88h16m35.37243935s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.506575192 (88h16m35.377658238s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.511021099 (88h16m35.382104134s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.519588893 (88h16m35.390671938s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.540564915 (88h16m35.41164795s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.555394233 (88h16m35.426477278s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.565337939 (88h16m35.436420984s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.576688943 (88h16m35.447771978s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.589009313 (88h16m35.460092368s) [noise=true , address=45.91.116.13:4181 , piece_id=X6IEEUKM5GDXZD2URW6YAPSN4DXWOCD2NK56K5PFFAYEHL2RTJUQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619301786 (88h16m35.490384821s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619323929 (88h16m35.490406964s) [seg_pos={0 0}, upload=77044]: Segment reached optimal threshold of 14 pieces. Dec 9 07:45:35.619554758 (88h16m35.490637793s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619564476 (88h16m35.490647531s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619577912 (88h16m35.490660947s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619583001 (88h16m35.490666037s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619751342 (88h16m35.490834377s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619757253 (88h16m35.490840298s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619853827 (88h16m35.490936862s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection Dec 9 07:45:35.619862484 (88h16m35.490945519s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619883484 (88h16m35.490966519s) [seg_pos={0 0}, upload=77044]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:45:35.619956242 (88h16m35.491039288s) [upload=77044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:45:35.619961382 (88h16m35.491044417s) [upload=77044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:47:56.102293256 (88h18m55.973376291s) [upload=77044]: Done uploading segments... Dec 9 07:47:56.102346187 (88h18m55.973429222s) [upload=77044]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.281158787s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.108757471s trace: https://jaeger.dev.storj.io/trace/50f433cc1e60731c FAIL (2025-12-09 08:08:35 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.108814459s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.005212259s trace: https://jaeger.dev.storj.io/trace/55634d9c1d55ff67 FAIL (2025-12-09 08:08:34 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.005245764s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.XXX........................................................] duration: 2.163259514s trace: https://jaeger.dev.storj.io/trace/178b4cba623ae119 FAIL (2025-12-09 08:08:31 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.163286746s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.......XX..X.......X...X..................X.............X...] duration: 5.828024679s trace: https://jaeger.dev.storj.io/trace/2e6e846a5a22deeb FAIL (2025-12-09 08:08:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.828074214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [XXX.X.......................................................] duration: 6.746176178s trace: https://jaeger.dev.storj.io/trace/34e772a771ff36b2 FAIL (2025-12-09 08:08:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :29:22.029442363 (88h0m21.900525398s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.029471198 (88h0m21.900554233s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.030555563 (88h0m21.901638608s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031156617 (88h0m21.902239662s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031594381 (88h0m21.902677416s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032034670 (88h0m21.903117715s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032167963 (88h0m21.903250998s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032800087 (88h0m21.903883132s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:23.836496806 (88h0m23.707579841s) [upload=76810]: Next returned nil segment Dec 9 07:29:23.836512486 (88h0m23.707595531s) [upload=76810]: Waiting for error group managing segments... Dec 9 07:29:24.124325638 (88h0m23.995408683s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.114522896 (88h0m24.985605941s) [noise=true , address=145.79.227.21:4475 , piece_id=ED4VS3YEIFPJGGKW36BJKVLT3EXGVOFG3SD23LEPPFIAOOKHUD4Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.377083973 (88h0m25.248167008s) [noise=true , address=145.79.227.41:4086 , piece_id=EZUBNMEUWTNJBXJBWJYXFJQEPFFVQOUHZAWVJW6NCWKGRKAS2PSA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.480466423 (88h0m25.351549468s) [noise=true , address=145.79.227.42:4121 , piece_id=7I2TXNXZM2KM5TEDDF7Q5DDUSBAPBE3WWFTTJFGNM5JVVCTQNU5Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:26.767037853 (88h0m26.638120908s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:49.689337786 (88h0m49.560420841s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection Dec 9 07:29:49.689500304 (88h0m49.56058335s) [noise=true , address=145.79.227.34:4907 , piece_id=UG6M3T655YGW4OGBYEVOLQVR7U5XWXPDDNXBZAPTHYP2DYWDRRYA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection Dec 9 07:29:49.689541303 (88h0m49.560624348s) [noise=true , address=145.79.227.33:4870 , piece_id=6Q5XSDCJCDRWHPO7JWCDNPQM5JN2ESDCRXWY5V23AGA6Q4C5Y7HQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection Dec 9 07:29:49.689654439 (88h0m49.560737474s) [noise=true , address=145.79.227.29:4943 , piece_id=FQ2FNNQR6QBWTJOIQBIOU7BSWIA6ZZMDWC4DE42NUNR5Z7ZDGCGQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection Dec 9 07:29:49.689700015 (88h0m49.56078306s) [noise=true , address=145.79.227.42:4122 , piece_id=5WKTAWBKWCZVU3YX4RSF4FQHQABNCZUHTXMTK3AIHUAGYDQXQYVQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection Dec 9 07:29:49.689861212 (88h0m49.560944247s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection Dec 9 07:29:49.689914644 (88h0m49.560997689s) [noise=true , address=145.79.227.28:4725 , piece_id=AGDXVUHDFM7DC2HNW3YUKREAROODLZBIVUMTFYBY3CXYQ7HSYGWA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection Dec 9 07:29:49.690003323 (88h0m49.561086368s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection Dec 9 07:29:49.690090949 (88h0m49.561173994s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection Dec 9 07:29:49.690262546 (88h0m49.561345591s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection Dec 9 07:29:49.690582726 (88h0m49.561665772s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:50466->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:29:49.690631769 (88h0m49.561714805s) [seg_pos={0 0}, upload=76810]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Dec 9 07:29:50.737500571 (88h0m50.608583616s) [upload=76810]: Done uploading segments... Dec 9 07:29:50.737550175 (88h0m50.60863321s) [upload=76810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.746198251s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.16742279s trace: https://jaeger.dev.storj.io/trace/7014f415ea72bfa2 FAIL (2025-12-09 08:08:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:56:08.451608077 (88h27m8.322691112s) [upload=77198]: Starting upload Dec 9 07:56:08.451664594 (88h27m8.322747639s) [upload=77198]: Done starting upload Dec 9 07:56:08.451743716 (88h27m8.322826761s) [upload=77198]: Uploading segments... Dec 9 07:56:08.451810052 (88h27m8.322893127s) [upload=77198]: Got next segment. Inline: true Dec 9 07:56:08.451818759 (88h27m8.322901804s) [upload=77198]: Waiting for error group managing segments... Dec 9 07:56:08.451893511 (88h27m8.322976546s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:56:08.451899602 (88h27m8.322982647s) [upload=77198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:56:08.451906044 (88h27m8.322989089s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:57:04.013681009 (88h28m3.884764054s) [upload=77198]: Done uploading segments... Dec 9 07:57:04.013737777 (88h28m3.884820822s) [upload=77198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.167448519s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.....X.XX.XX..X..................X..........................] duration: 3.47441034s trace: https://jaeger.dev.storj.io/trace/610b5afc07f67951 FAIL (2025-12-09 08:08:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.474474754s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X....................X............................] duration: 8.707907454s trace: https://jaeger.dev.storj.io/trace/7fe88be41ea8781e FAIL (2025-12-09 08:08:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng piece... Dec 9 07:45:33.502533729 (88h16m33.373616774s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503057677 (88h16m33.374140712s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503561135 (88h16m33.37464417s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504037222 (88h16m33.375120257s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504531994 (88h16m33.375615039s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505014954 (88h16m33.376098029s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505576262 (88h16m33.376659307s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.506942875 (88h16m33.37802591s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508063960 (88h16m33.379147005s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508207313 (88h16m33.379290358s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.509568444 (88h16m33.38065149s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.511408599 (88h16m33.382491644s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.513502646 (88h16m33.384585681s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.515541418 (88h16m33.386624484s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.517269299 (88h16m33.388352334s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.518893461 (88h16m33.389976496s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:34.970483840 (88h16m34.841566885s) [upload=77044]: Next returned nil segment Dec 9 07:45:34.970497998 (88h16m34.841581043s) [upload=77044]: Waiting for error group managing segments... Dec 9 07:45:35.375513876 (88h16m35.246596921s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.378851702 (88h16m35.249934747s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.452093509 (88h16m35.323176544s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.484823964 (88h16m35.355906999s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.501356315 (88h16m35.37243935s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.506575192 (88h16m35.377658238s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.511021099 (88h16m35.382104134s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.519588893 (88h16m35.390671938s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.540564915 (88h16m35.41164795s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.555394233 (88h16m35.426477278s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.565337939 (88h16m35.436420984s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.576688943 (88h16m35.447771978s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.589009313 (88h16m35.460092368s) [noise=true , address=45.91.116.13:4181 , piece_id=X6IEEUKM5GDXZD2URW6YAPSN4DXWOCD2NK56K5PFFAYEHL2RTJUQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619301786 (88h16m35.490384821s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619323929 (88h16m35.490406964s) [seg_pos={0 0}, upload=77044]: Segment reached optimal threshold of 14 pieces. Dec 9 07:45:35.619554758 (88h16m35.490637793s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619564476 (88h16m35.490647531s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619577912 (88h16m35.490660947s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619583001 (88h16m35.490666037s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619751342 (88h16m35.490834377s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619757253 (88h16m35.490840298s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619853827 (88h16m35.490936862s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection Dec 9 07:45:35.619862484 (88h16m35.490945519s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619883484 (88h16m35.490966519s) [seg_pos={0 0}, upload=77044]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:45:35.619956242 (88h16m35.491039288s) [upload=77044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:45:35.619961382 (88h16m35.491044417s) [upload=77044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:47:56.102293256 (88h18m55.973376291s) [upload=77044]: Done uploading segments... Dec 9 07:47:56.102346187 (88h18m55.973429222s) [upload=77044]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.707939845s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..XXX.......................................................] duration: 2.295722709s trace: https://jaeger.dev.storj.io/trace/178b4cba623ae119 FAIL (2025-12-09 08:08:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.295747356s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 3.160251204s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/43834d9fac36e837 FAIL (2025-12-09 08:08:12 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 3.160284586s core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 3.216648231s trace: https://jaeger.dev.storj.io/trace/42dcab7a5e240fd8 FAIL (2025-12-09 08:08:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:28:09.412200246 (87h59m9.283283281s) [upload=76792]: Starting upload Dec 9 07:28:09.412257335 (87h59m9.283340371s) [upload=76792]: Done starting upload Dec 9 07:28:09.412337338 (87h59m9.283420383s) [upload=76792]: Uploading segments... Dec 9 07:28:09.412384597 (87h59m9.283467632s) [upload=76792]: Got next segment. Inline: true Dec 9 07:28:09.412398334 (87h59m9.283481389s) [upload=76792]: Waiting for error group managing segments... Dec 9 07:28:09.412446225 (87h59m9.28352927s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:28:09.412452456 (87h59m9.283535491s) [upload=76792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:28:09.412455432 (87h59m9.283538477s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:28:19.867794892 (87h59m19.738877927s) [upload=76792]: Done uploading segments... Dec 9 07:28:19.867843895 (87h59m19.73892694s) [upload=76792]: Upload finished. err: metaclient: context deadline exceeded Duration:* 3.216680292s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.516042038s trace: https://jaeger.dev.storj.io/trace/7014f415ea72bfa2 FAIL (2025-12-09 08:08:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:56:08.451608077 (88h27m8.322691112s) [upload=77198]: Starting upload Dec 9 07:56:08.451664594 (88h27m8.322747639s) [upload=77198]: Done starting upload Dec 9 07:56:08.451743716 (88h27m8.322826761s) [upload=77198]: Uploading segments... Dec 9 07:56:08.451810052 (88h27m8.322893127s) [upload=77198]: Got next segment. Inline: true Dec 9 07:56:08.451818759 (88h27m8.322901804s) [upload=77198]: Waiting for error group managing segments... Dec 9 07:56:08.451893511 (88h27m8.322976546s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:56:08.451899602 (88h27m8.322982647s) [upload=77198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:56:08.451906044 (88h27m8.322989089s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:57:04.013681009 (88h28m3.884764054s) [upload=77198]: Done uploading segments... Dec 9 07:57:04.013737777 (88h28m3.884820822s) [upload=77198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.516073348s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 5.950108845s trace: https://jaeger.dev.storj.io/trace/5f10a1154ce06783 FAIL (2025-12-09 08:07:40 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.950140926s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 484.870028ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt trace: https://jaeger.dev.storj.io/trace/51a280a8635ce0b9 FAIL (2025-12-09 08:07:39 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 484.895838ms edge linksharing us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [......X.XX.XX..X..................X.........................] duration: 3.349601093s trace: https://jaeger.dev.storj.io/trace/610b5afc07f67951 FAIL (2025-12-09 08:07:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.349624267s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.XXX.X......................................................] duration: 5.775256746s trace: https://jaeger.dev.storj.io/trace/34e772a771ff36b2 FAIL (2025-12-09 08:07:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :29:22.029442363 (88h0m21.900525398s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.029471198 (88h0m21.900554233s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.030555563 (88h0m21.901638608s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031156617 (88h0m21.902239662s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031594381 (88h0m21.902677416s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032034670 (88h0m21.903117715s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032167963 (88h0m21.903250998s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032800087 (88h0m21.903883132s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:23.836496806 (88h0m23.707579841s) [upload=76810]: Next returned nil segment Dec 9 07:29:23.836512486 (88h0m23.707595531s) [upload=76810]: Waiting for error group managing segments... Dec 9 07:29:24.124325638 (88h0m23.995408683s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.114522896 (88h0m24.985605941s) [noise=true , address=145.79.227.21:4475 , piece_id=ED4VS3YEIFPJGGKW36BJKVLT3EXGVOFG3SD23LEPPFIAOOKHUD4Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.377083973 (88h0m25.248167008s) [noise=true , address=145.79.227.41:4086 , piece_id=EZUBNMEUWTNJBXJBWJYXFJQEPFFVQOUHZAWVJW6NCWKGRKAS2PSA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.480466423 (88h0m25.351549468s) [noise=true , address=145.79.227.42:4121 , piece_id=7I2TXNXZM2KM5TEDDF7Q5DDUSBAPBE3WWFTTJFGNM5JVVCTQNU5Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:26.767037853 (88h0m26.638120908s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:49.689337786 (88h0m49.560420841s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection Dec 9 07:29:49.689500304 (88h0m49.56058335s) [noise=true , address=145.79.227.34:4907 , piece_id=UG6M3T655YGW4OGBYEVOLQVR7U5XWXPDDNXBZAPTHYP2DYWDRRYA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection Dec 9 07:29:49.689541303 (88h0m49.560624348s) [noise=true , address=145.79.227.33:4870 , piece_id=6Q5XSDCJCDRWHPO7JWCDNPQM5JN2ESDCRXWY5V23AGA6Q4C5Y7HQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection Dec 9 07:29:49.689654439 (88h0m49.560737474s) [noise=true , address=145.79.227.29:4943 , piece_id=FQ2FNNQR6QBWTJOIQBIOU7BSWIA6ZZMDWC4DE42NUNR5Z7ZDGCGQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection Dec 9 07:29:49.689700015 (88h0m49.56078306s) [noise=true , address=145.79.227.42:4122 , piece_id=5WKTAWBKWCZVU3YX4RSF4FQHQABNCZUHTXMTK3AIHUAGYDQXQYVQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection Dec 9 07:29:49.689861212 (88h0m49.560944247s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection Dec 9 07:29:49.689914644 (88h0m49.560997689s) [noise=true , address=145.79.227.28:4725 , piece_id=AGDXVUHDFM7DC2HNW3YUKREAROODLZBIVUMTFYBY3CXYQ7HSYGWA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection Dec 9 07:29:49.690003323 (88h0m49.561086368s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection Dec 9 07:29:49.690090949 (88h0m49.561173994s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection Dec 9 07:29:49.690262546 (88h0m49.561345591s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection Dec 9 07:29:49.690582726 (88h0m49.561665772s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:50466->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:29:49.690631769 (88h0m49.561714805s) [seg_pos={0 0}, upload=76810]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Dec 9 07:29:50.737500571 (88h0m50.608583616s) [upload=76810]: Done uploading segments... Dec 9 07:29:50.737550175 (88h0m50.60863321s) [upload=76810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.775289898s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...XXX......................................................] duration: 2.157141131s trace: https://jaeger.dev.storj.io/trace/178b4cba623ae119 FAIL (2025-12-09 08:07:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.157167722s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.048728078s trace: https://jaeger.dev.storj.io/trace/50f433cc1e60731c FAIL (2025-12-09 08:07:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.04875616s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.028358877s trace: https://jaeger.dev.storj.io/trace/55634d9c1d55ff67 FAIL (2025-12-09 08:07:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.028386038s core uplink us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 695.170872ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/43834d9fac36e837 FAIL (2025-12-09 08:07:26 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 695.328882ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [....XXX.....................................................] duration: 2.21268158s trace: https://jaeger.dev.storj.io/trace/178b4cba623ae119 FAIL (2025-12-09 08:07:17 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.212707169s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [........XX..X.......X...X..................X.............X..] duration: 7.337947339s trace: https://jaeger.dev.storj.io/trace/2e6e846a5a22deeb FAIL (2025-12-09 08:07:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.337976786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X....................X...........................] duration: 6.593869398s trace: https://jaeger.dev.storj.io/trace/7fe88be41ea8781e FAIL (2025-12-09 08:07:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ng piece... Dec 9 07:45:33.502533729 (88h16m33.373616774s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503057677 (88h16m33.374140712s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.503561135 (88h16m33.37464417s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504037222 (88h16m33.375120257s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.504531994 (88h16m33.375615039s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505014954 (88h16m33.376098029s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.505576262 (88h16m33.376659307s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.506942875 (88h16m33.37802591s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508063960 (88h16m33.379147005s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.508207313 (88h16m33.379290358s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.509568444 (88h16m33.38065149s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.511408599 (88h16m33.382491644s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.513502646 (88h16m33.384585681s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.515541418 (88h16m33.386624484s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.517269299 (88h16m33.388352334s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:33.518893461 (88h16m33.389976496s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Uploading piece... Dec 9 07:45:34.970483840 (88h16m34.841566885s) [upload=77044]: Next returned nil segment Dec 9 07:45:34.970497998 (88h16m34.841581043s) [upload=77044]: Waiting for error group managing segments... Dec 9 07:45:35.375513876 (88h16m35.246596921s) [noise=true , address=145.79.227.23:4565 , piece_id=PRBWTR7SVNCCCP5ERNUPB6TJOPXSPYFSZJIOL3RLPC4KRVORYWXQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.378851702 (88h16m35.249934747s) [noise=true , address=145.79.227.52:4452 , piece_id=X2RBQGFCIHR4JAJTJJ3FX4CA6PQIQUA2YHYJUANV7JIU5PKAVNLA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.452093509 (88h16m35.323176544s) [noise=true , address=77.90.31.21:4895 , piece_id=SW37FGLFYCZCFUVZL7BISFI3FQS2G2RXXYLD2IPOH75UE475HCTQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.484823964 (88h16m35.355906999s) [noise=true , address=77.90.31.19:4875 , piece_id=EA3XPU4W6DRDZMYWY2XW4IQ35DMCQEV7XQSNHJN33H24K7MJVOKA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.501356315 (88h16m35.37243935s) [noise=true , address=77.90.31.16:4749 , piece_id=B7JNUSR74QRRNXEFUYILTIMALV5DN75TX7P67VQ6TDGXLBLFDCGQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.506575192 (88h16m35.377658238s) [noise=true , address=77.90.31.14:4683 , piece_id=FAS45YECJULRHU63VG4NFL3NPFCZPQW7IBELFWGVGMIE7YQS3J6Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.511021099 (88h16m35.382104134s) [noise=true , address=77.90.31.17:4797 , piece_id=XQINPPZJLK6HMR33M2SGC3EHJKL4XX7KZRWNM532PZHD3DEQ4M3Q, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.519588893 (88h16m35.390671938s) [noise=true , address=77.90.31.11:4580 , piece_id=UDQZXQNUWDEILKNR42DLKTYVIJX3M7Q2YH3KOCBETUEBQZZ2WU4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.540564915 (88h16m35.41164795s) [noise=true , address=45.91.116.11:4133 , piece_id=V72CQI43EBT2QWQKAC5GU2MSB7UKF7G5NIGMTLS7EOLOBGEG4YQQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.555394233 (88h16m35.426477278s) [noise=true , address=77.90.31.10:4546 , piece_id=SNWHQLYV5CBLDDT2BHM7IKY25CHDCJFTTT2APV2IN7RQQIVG7TZA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.565337939 (88h16m35.436420984s) [noise=true , address=45.91.116.8:4071 , piece_id=INHPP6K7NOVPOV2G6CAUFSIKF2Q3KB5SOEE53MVU4MUDPMLSLC4A, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.576688943 (88h16m35.447771978s) [noise=true , address=77.90.31.15:4732 , piece_id=HYM7NBL4TFURODTBMWBJ7J23DON4WNYFJQ4Q3WNEBN7KCZYV3ORA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.589009313 (88h16m35.460092368s) [noise=true , address=45.91.116.13:4181 , piece_id=X6IEEUKM5GDXZD2URW6YAPSN4DXWOCD2NK56K5PFFAYEHL2RTJUQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619301786 (88h16m35.490384821s) [noise=true , address=45.91.116.6:4987 , piece_id=JEQK2JKSRMQLGLNLVPBCPS6MCRYOJ7W7IEB4HBTBYFLIGO3CRXKQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: <nil> Dec 9 07:45:35.619323929 (88h16m35.490406964s) [seg_pos={0 0}, upload=77044]: Segment reached optimal threshold of 14 pieces. Dec 9 07:45:35.619554758 (88h16m35.490637793s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NmgZdLWKjrDMmNcgmnzH475akX4WtA9zGTLxZgVW7JpaecBjq): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619564476 (88h16m35.490647531s) [noise=true , address=45.91.116.12:4231 , piece_id=HZ7JTNUJBWREEUHR3A6D5GG5ENKCA4MXAHCCNMI4MU6ZHKZUC6MQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619577912 (88h16m35.490660947s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619583001 (88h16m35.490666037s) [noise=true , address=145.79.227.30:4796 , piece_id=I25OEVIWGQMZHFZMJMP7H5BMET6GX72RAEBEKV2ZSGQHQGP6QNTA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619751342 (88h16m35.490834377s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo): protocol: expected piece hash; context canceled; EOF Dec 9 07:45:35.619757253 (88h16m35.490840298s) [noise=true , address=77.90.31.18:4827 , piece_id=UK3RJZ5DHNVWCMTUWGTAEJJCY7YXK7EWSQKGEQYV4JYFZ6AV3FEQ, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619853827 (88h16m35.490936862s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34936->64.49.13.169:4707: use of closed network connection Dec 9 07:45:35.619862484 (88h16m35.490945519s) [noise=true , address=64.49.13.169:4707 , piece_id=43LJJAVA7GKYJEUEYARA5BD7VYVAC3HOH3PFOWFTJEYZXDANCAVA, seg_pos={0 0}, upload=77044]: Long tail cancellation due to optimal threshold Dec 9 07:45:35.619883484 (88h16m35.490966519s) [seg_pos={0 0}, upload=77044]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:45:35.619956242 (88h16m35.491039288s) [upload=77044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:45:35.619961382 (88h16m35.491044417s) [upload=77044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:47:56.102293256 (88h18m55.973376291s) [upload=77044]: Done uploading segments... Dec 9 07:47:56.102346187 (88h18m55.973429222s) [upload=77044]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.59389188s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 5.943390815s trace: https://jaeger.dev.storj.io/trace/5f10a1154ce06783 FAIL (2025-12-09 08:07:08 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.943431954s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.229747499s trace: https://jaeger.dev.storj.io/trace/42dcab7a5e240fd8 FAIL (2025-12-09 08:07:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:28:09.412200246 (87h59m9.283283281s) [upload=76792]: Starting upload Dec 9 07:28:09.412257335 (87h59m9.283340371s) [upload=76792]: Done starting upload Dec 9 07:28:09.412337338 (87h59m9.283420383s) [upload=76792]: Uploading segments... Dec 9 07:28:09.412384597 (87h59m9.283467632s) [upload=76792]: Got next segment. Inline: true Dec 9 07:28:09.412398334 (87h59m9.283481389s) [upload=76792]: Waiting for error group managing segments... Dec 9 07:28:09.412446225 (87h59m9.28352927s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:28:09.412452456 (87h59m9.283535491s) [upload=76792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:28:09.412455432 (87h59m9.283538477s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:28:19.867794892 (87h59m19.738877927s) [upload=76792]: Done uploading segments... Dec 9 07:28:19.867843895 (87h59m19.73892694s) [upload=76792]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.229778016s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..XXX.X.....................................................] duration: 6.348372663s trace: https://jaeger.dev.storj.io/trace/34e772a771ff36b2 FAIL (2025-12-09 08:07:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :29:22.029442363 (88h0m21.900525398s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.029471198 (88h0m21.900554233s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.030555563 (88h0m21.901638608s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031156617 (88h0m21.902239662s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031594381 (88h0m21.902677416s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032034670 (88h0m21.903117715s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032167963 (88h0m21.903250998s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032800087 (88h0m21.903883132s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:23.836496806 (88h0m23.707579841s) [upload=76810]: Next returned nil segment Dec 9 07:29:23.836512486 (88h0m23.707595531s) [upload=76810]: Waiting for error group managing segments... Dec 9 07:29:24.124325638 (88h0m23.995408683s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.114522896 (88h0m24.985605941s) [noise=true , address=145.79.227.21:4475 , piece_id=ED4VS3YEIFPJGGKW36BJKVLT3EXGVOFG3SD23LEPPFIAOOKHUD4Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.377083973 (88h0m25.248167008s) [noise=true , address=145.79.227.41:4086 , piece_id=EZUBNMEUWTNJBXJBWJYXFJQEPFFVQOUHZAWVJW6NCWKGRKAS2PSA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.480466423 (88h0m25.351549468s) [noise=true , address=145.79.227.42:4121 , piece_id=7I2TXNXZM2KM5TEDDF7Q5DDUSBAPBE3WWFTTJFGNM5JVVCTQNU5Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:26.767037853 (88h0m26.638120908s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:49.689337786 (88h0m49.560420841s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection Dec 9 07:29:49.689500304 (88h0m49.56058335s) [noise=true , address=145.79.227.34:4907 , piece_id=UG6M3T655YGW4OGBYEVOLQVR7U5XWXPDDNXBZAPTHYP2DYWDRRYA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection Dec 9 07:29:49.689541303 (88h0m49.560624348s) [noise=true , address=145.79.227.33:4870 , piece_id=6Q5XSDCJCDRWHPO7JWCDNPQM5JN2ESDCRXWY5V23AGA6Q4C5Y7HQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection Dec 9 07:29:49.689654439 (88h0m49.560737474s) [noise=true , address=145.79.227.29:4943 , piece_id=FQ2FNNQR6QBWTJOIQBIOU7BSWIA6ZZMDWC4DE42NUNR5Z7ZDGCGQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection Dec 9 07:29:49.689700015 (88h0m49.56078306s) [noise=true , address=145.79.227.42:4122 , piece_id=5WKTAWBKWCZVU3YX4RSF4FQHQABNCZUHTXMTK3AIHUAGYDQXQYVQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection Dec 9 07:29:49.689861212 (88h0m49.560944247s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection Dec 9 07:29:49.689914644 (88h0m49.560997689s) [noise=true , address=145.79.227.28:4725 , piece_id=AGDXVUHDFM7DC2HNW3YUKREAROODLZBIVUMTFYBY3CXYQ7HSYGWA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection Dec 9 07:29:49.690003323 (88h0m49.561086368s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection Dec 9 07:29:49.690090949 (88h0m49.561173994s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection Dec 9 07:29:49.690262546 (88h0m49.561345591s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection Dec 9 07:29:49.690582726 (88h0m49.561665772s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:50466->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:29:49.690631769 (88h0m49.561714805s) [seg_pos={0 0}, upload=76810]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Dec 9 07:29:50.737500571 (88h0m50.608583616s) [upload=76810]: Done uploading segments... Dec 9 07:29:50.737550175 (88h0m50.60863321s) [upload=76810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 6.348404614s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 2 out of 60 is failed (threshold=3) [X..X........................................................] duration: 708.257832ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/43834d9fac36e837 FAIL (2025-12-09 08:07:01 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 708.287489ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.104505034s trace: https://jaeger.dev.storj.io/trace/7014f415ea72bfa2 FAIL (2025-12-09 08:06:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:56:08.451608077 (88h27m8.322691112s) [upload=77198]: Starting upload Dec 9 07:56:08.451664594 (88h27m8.322747639s) [upload=77198]: Done starting upload Dec 9 07:56:08.451743716 (88h27m8.322826761s) [upload=77198]: Uploading segments... Dec 9 07:56:08.451810052 (88h27m8.322893127s) [upload=77198]: Got next segment. Inline: true Dec 9 07:56:08.451818759 (88h27m8.322901804s) [upload=77198]: Waiting for error group managing segments... Dec 9 07:56:08.451893511 (88h27m8.322976546s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:56:08.451899602 (88h27m8.322982647s) [upload=77198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:56:08.451906044 (88h27m8.322989089s) [upload=77198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:57:04.013681009 (88h28m3.884764054s) [upload=77198]: Done uploading segments... Dec 9 07:57:04.013737777 (88h28m3.884820822s) [upload=77198]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.104553444s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.........XX..X.......X...X..................X.............X.] duration: 5.611312014s trace: https://jaeger.dev.storj.io/trace/2e6e846a5a22deeb FAIL (2025-12-09 08:06:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.611342773s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.003563718s trace: https://jaeger.dev.storj.io/trace/50f433cc1e60731c FAIL (2025-12-09 08:06:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.003590711s core uplink eu1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)