Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 868255 reports (Page 11630 of 17366)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....................................X.XX.XX..X..............] duration: 3.5258487s trace: https://jaeger.dev.storj.io/trace/402bfb4648c7d69b FAIL (2025-12-09 07:44:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.52587487s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...................................XXX......................] duration: 2.249648077s trace: https://jaeger.dev.storj.io/trace/178b4cba623ae119 FAIL (2025-12-09 07:44:44 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.24968136s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 3 out of 60 is failed (threshold=3) [....................................XX..X...................] duration: 691.377717ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/43834d9fac36e837 FAIL (2025-12-09 07:44:41 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 691.404839ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................XXX.X......................] duration: 5.669734837s trace: https://jaeger.dev.storj.io/trace/34e772a771ff36b2 FAIL (2025-12-09 07:44:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :29:22.029442363 (88h0m21.900525398s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.029471198 (88h0m21.900554233s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.030555563 (88h0m21.901638608s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031156617 (88h0m21.902239662s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031594381 (88h0m21.902677416s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032034670 (88h0m21.903117715s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032167963 (88h0m21.903250998s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032800087 (88h0m21.903883132s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:23.836496806 (88h0m23.707579841s) [upload=76810]: Next returned nil segment Dec 9 07:29:23.836512486 (88h0m23.707595531s) [upload=76810]: Waiting for error group managing segments... Dec 9 07:29:24.124325638 (88h0m23.995408683s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.114522896 (88h0m24.985605941s) [noise=true , address=145.79.227.21:4475 , piece_id=ED4VS3YEIFPJGGKW36BJKVLT3EXGVOFG3SD23LEPPFIAOOKHUD4Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.377083973 (88h0m25.248167008s) [noise=true , address=145.79.227.41:4086 , piece_id=EZUBNMEUWTNJBXJBWJYXFJQEPFFVQOUHZAWVJW6NCWKGRKAS2PSA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.480466423 (88h0m25.351549468s) [noise=true , address=145.79.227.42:4121 , piece_id=7I2TXNXZM2KM5TEDDF7Q5DDUSBAPBE3WWFTTJFGNM5JVVCTQNU5Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:26.767037853 (88h0m26.638120908s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:49.689337786 (88h0m49.560420841s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection Dec 9 07:29:49.689500304 (88h0m49.56058335s) [noise=true , address=145.79.227.34:4907 , piece_id=UG6M3T655YGW4OGBYEVOLQVR7U5XWXPDDNXBZAPTHYP2DYWDRRYA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection Dec 9 07:29:49.689541303 (88h0m49.560624348s) [noise=true , address=145.79.227.33:4870 , piece_id=6Q5XSDCJCDRWHPO7JWCDNPQM5JN2ESDCRXWY5V23AGA6Q4C5Y7HQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection Dec 9 07:29:49.689654439 (88h0m49.560737474s) [noise=true , address=145.79.227.29:4943 , piece_id=FQ2FNNQR6QBWTJOIQBIOU7BSWIA6ZZMDWC4DE42NUNR5Z7ZDGCGQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection Dec 9 07:29:49.689700015 (88h0m49.56078306s) [noise=true , address=145.79.227.42:4122 , piece_id=5WKTAWBKWCZVU3YX4RSF4FQHQABNCZUHTXMTK3AIHUAGYDQXQYVQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection Dec 9 07:29:49.689861212 (88h0m49.560944247s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection Dec 9 07:29:49.689914644 (88h0m49.560997689s) [noise=true , address=145.79.227.28:4725 , piece_id=AGDXVUHDFM7DC2HNW3YUKREAROODLZBIVUMTFYBY3CXYQ7HSYGWA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection Dec 9 07:29:49.690003323 (88h0m49.561086368s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection Dec 9 07:29:49.690090949 (88h0m49.561173994s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection Dec 9 07:29:49.690262546 (88h0m49.561345591s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection Dec 9 07:29:49.690582726 (88h0m49.561665772s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:50466->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:29:49.690631769 (88h0m49.561714805s) [seg_pos={0 0}, upload=76810]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Dec 9 07:29:50.737500571 (88h0m50.608583616s) [upload=76810]: Done uploading segments... Dec 9 07:29:50.737550175 (88h0m50.60863321s) [upload=76810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.669790293s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..........................X...........XX..X.......X...X.....] duration: 5.812109646s trace: https://jaeger.dev.storj.io/trace/76e27d463b59cd34 FAIL (2025-12-09 07:44:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.812140205s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.703414609s trace: https://jaeger.dev.storj.io/trace/219dd7469694ce8d FAIL (2025-12-09 07:44:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.221678783 (87h58m24.092761818s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222184665 (87h58m24.09326771s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222790810 (87h58m24.093873855s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223285452 (87h58m24.094368487s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223348001 (87h58m24.094431036s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223906905 (87h58m24.09498994s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224273312 (87h58m24.095356357s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224668005 (87h58m24.09575104s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225016438 (87h58m24.096099483s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225398876 (87h58m24.096481911s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225880544 (87h58m24.096963579s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.226416704 (87h58m24.097499739s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.227984600 (87h58m24.099067635s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.229456091 (87h58m24.100539136s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.231011924 (87h58m24.102094969s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.232660333 (87h58m24.103743388s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:25.694926842 (87h58m25.566009888s) [upload=76783]: Next returned nil segment Dec 9 07:27:25.694949827 (87h58m25.566032862s) [upload=76783]: Waiting for error group managing segments... Dec 9 07:27:26.134631394 (87h58m26.00571443s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.144638911 (87h58m26.015721956s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.194703193 (87h58m26.065786228s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.202448432 (87h58m26.073531477s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.218793957 (87h58m26.089876992s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.219708348 (87h58m26.090791393s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.227913963 (87h58m26.098996998s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.238868943 (87h58m26.109951988s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.249699476 (87h58m26.120782522s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.263815877 (87h58m26.134898912s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.264418876 (87h58m26.135501911s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.280320365 (87h58m26.1514034s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.294136013 (87h58m26.165219058s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321886986 (87h58m26.192970031s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321911793 (87h58m26.192994838s) [seg_pos={0 0}, upload=76783]: Segment reached optimal threshold of 14 pieces. Dec 9 07:27:26.322167190 (87h58m26.193250235s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322175145 (87h58m26.19325821s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.322770298 (87h58m26.193853333s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322779535 (87h58m26.19386258s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323079737 (87h58m26.194162792s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT): protocol: read tcp 188.166.225.170:43106->145.79.227.36:4548: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323089335 (87h58m26.19417238s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323372955 (87h58m26.19445599s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: read tcp 188.166.225.170:50502->64.49.13.169:4716: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323381462 (87h58m26.194464507s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323400197 (87h58m26.194483242s) [seg_pos={0 0}, upload=76783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:27:26.323490970 (87h58m26.194574015s) [upload=76783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:27:26.323497242 (87h58m26.194580287s) [upload=76783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:30:43.052543346 (88h1m42.923626391s) [upload=76783]: Done uploading segments... Dec 9 07:30:43.052668744 (88h1m42.923751799s) [upload=76783]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.703458363s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.................................XXX.X.....................] duration: 5.626049666s trace: https://jaeger.dev.storj.io/trace/34e772a771ff36b2 FAIL (2025-12-09 07:44:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :29:22.029442363 (88h0m21.900525398s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.029471198 (88h0m21.900554233s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.030555563 (88h0m21.901638608s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031156617 (88h0m21.902239662s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031594381 (88h0m21.902677416s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032034670 (88h0m21.903117715s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032167963 (88h0m21.903250998s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032800087 (88h0m21.903883132s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:23.836496806 (88h0m23.707579841s) [upload=76810]: Next returned nil segment Dec 9 07:29:23.836512486 (88h0m23.707595531s) [upload=76810]: Waiting for error group managing segments... Dec 9 07:29:24.124325638 (88h0m23.995408683s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.114522896 (88h0m24.985605941s) [noise=true , address=145.79.227.21:4475 , piece_id=ED4VS3YEIFPJGGKW36BJKVLT3EXGVOFG3SD23LEPPFIAOOKHUD4Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.377083973 (88h0m25.248167008s) [noise=true , address=145.79.227.41:4086 , piece_id=EZUBNMEUWTNJBXJBWJYXFJQEPFFVQOUHZAWVJW6NCWKGRKAS2PSA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.480466423 (88h0m25.351549468s) [noise=true , address=145.79.227.42:4121 , piece_id=7I2TXNXZM2KM5TEDDF7Q5DDUSBAPBE3WWFTTJFGNM5JVVCTQNU5Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:26.767037853 (88h0m26.638120908s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:49.689337786 (88h0m49.560420841s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection Dec 9 07:29:49.689500304 (88h0m49.56058335s) [noise=true , address=145.79.227.34:4907 , piece_id=UG6M3T655YGW4OGBYEVOLQVR7U5XWXPDDNXBZAPTHYP2DYWDRRYA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection Dec 9 07:29:49.689541303 (88h0m49.560624348s) [noise=true , address=145.79.227.33:4870 , piece_id=6Q5XSDCJCDRWHPO7JWCDNPQM5JN2ESDCRXWY5V23AGA6Q4C5Y7HQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection Dec 9 07:29:49.689654439 (88h0m49.560737474s) [noise=true , address=145.79.227.29:4943 , piece_id=FQ2FNNQR6QBWTJOIQBIOU7BSWIA6ZZMDWC4DE42NUNR5Z7ZDGCGQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection Dec 9 07:29:49.689700015 (88h0m49.56078306s) [noise=true , address=145.79.227.42:4122 , piece_id=5WKTAWBKWCZVU3YX4RSF4FQHQABNCZUHTXMTK3AIHUAGYDQXQYVQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection Dec 9 07:29:49.689861212 (88h0m49.560944247s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection Dec 9 07:29:49.689914644 (88h0m49.560997689s) [noise=true , address=145.79.227.28:4725 , piece_id=AGDXVUHDFM7DC2HNW3YUKREAROODLZBIVUMTFYBY3CXYQ7HSYGWA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection Dec 9 07:29:49.690003323 (88h0m49.561086368s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection Dec 9 07:29:49.690090949 (88h0m49.561173994s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection Dec 9 07:29:49.690262546 (88h0m49.561345591s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection Dec 9 07:29:49.690582726 (88h0m49.561665772s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:50466->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:29:49.690631769 (88h0m49.561714805s) [seg_pos={0 0}, upload=76810]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Dec 9 07:29:50.737500571 (88h0m50.608583616s) [upload=76810]: Done uploading segments... Dec 9 07:29:50.737550175 (88h0m50.60863321s) [upload=76810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.626079933s core uplink us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 471.547226ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt trace: https://jaeger.dev.storj.io/trace/51a280a8635ce0b9 FAIL (2025-12-09 07:44:13 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 471.572716ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.263158539s trace: https://jaeger.dev.storj.io/trace/50f433cc1e60731c FAIL (2025-12-09 07:44:12 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.263184458s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.....................................X.XX.XX..X.............] duration: 3.410889666s trace: https://jaeger.dev.storj.io/trace/402bfb4648c7d69b FAIL (2025-12-09 07:44:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.410972113s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.99463913s trace: https://jaeger.dev.storj.io/trace/42dcab7a5e240fd8 FAIL (2025-12-09 07:44:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:28:09.412200246 (87h59m9.283283281s) [upload=76792]: Starting upload Dec 9 07:28:09.412257335 (87h59m9.283340371s) [upload=76792]: Done starting upload Dec 9 07:28:09.412337338 (87h59m9.283420383s) [upload=76792]: Uploading segments... Dec 9 07:28:09.412384597 (87h59m9.283467632s) [upload=76792]: Got next segment. Inline: true Dec 9 07:28:09.412398334 (87h59m9.283481389s) [upload=76792]: Waiting for error group managing segments... Dec 9 07:28:09.412446225 (87h59m9.28352927s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:28:09.412452456 (87h59m9.283535491s) [upload=76792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:28:09.412455432 (87h59m9.283538477s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:28:19.867794892 (87h59m19.738877927s) [upload=76792]: Done uploading segments... Dec 9 07:28:19.867843895 (87h59m19.73892694s) [upload=76792]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.994667064s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X...................X...........................] duration: 5.824409055s trace: https://jaeger.dev.storj.io/trace/59c049b2e292a646 FAIL (2025-12-09 07:44:10 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.824438721s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 5.824647167s trace: https://jaeger.dev.storj.io/trace/5f10a1154ce06783 FAIL (2025-12-09 07:44:07 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.824699056s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 3 out of 60 is failed (threshold=3) [.....................................XX..X..................] duration: 686.624776ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/43834d9fac36e837 FAIL (2025-12-09 07:44:03 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 686.688488ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.................................XXX.X....................] duration: 17.009100472s trace: https://jaeger.dev.storj.io/trace/34e772a771ff36b2 FAIL (2025-12-09 07:43:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :29:22.029442363 (88h0m21.900525398s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.029471198 (88h0m21.900554233s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.030555563 (88h0m21.901638608s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031156617 (88h0m21.902239662s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031594381 (88h0m21.902677416s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032034670 (88h0m21.903117715s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032167963 (88h0m21.903250998s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032800087 (88h0m21.903883132s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:23.836496806 (88h0m23.707579841s) [upload=76810]: Next returned nil segment Dec 9 07:29:23.836512486 (88h0m23.707595531s) [upload=76810]: Waiting for error group managing segments... Dec 9 07:29:24.124325638 (88h0m23.995408683s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.114522896 (88h0m24.985605941s) [noise=true , address=145.79.227.21:4475 , piece_id=ED4VS3YEIFPJGGKW36BJKVLT3EXGVOFG3SD23LEPPFIAOOKHUD4Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.377083973 (88h0m25.248167008s) [noise=true , address=145.79.227.41:4086 , piece_id=EZUBNMEUWTNJBXJBWJYXFJQEPFFVQOUHZAWVJW6NCWKGRKAS2PSA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.480466423 (88h0m25.351549468s) [noise=true , address=145.79.227.42:4121 , piece_id=7I2TXNXZM2KM5TEDDF7Q5DDUSBAPBE3WWFTTJFGNM5JVVCTQNU5Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:26.767037853 (88h0m26.638120908s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:49.689337786 (88h0m49.560420841s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection Dec 9 07:29:49.689500304 (88h0m49.56058335s) [noise=true , address=145.79.227.34:4907 , piece_id=UG6M3T655YGW4OGBYEVOLQVR7U5XWXPDDNXBZAPTHYP2DYWDRRYA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection Dec 9 07:29:49.689541303 (88h0m49.560624348s) [noise=true , address=145.79.227.33:4870 , piece_id=6Q5XSDCJCDRWHPO7JWCDNPQM5JN2ESDCRXWY5V23AGA6Q4C5Y7HQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection Dec 9 07:29:49.689654439 (88h0m49.560737474s) [noise=true , address=145.79.227.29:4943 , piece_id=FQ2FNNQR6QBWTJOIQBIOU7BSWIA6ZZMDWC4DE42NUNR5Z7ZDGCGQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection Dec 9 07:29:49.689700015 (88h0m49.56078306s) [noise=true , address=145.79.227.42:4122 , piece_id=5WKTAWBKWCZVU3YX4RSF4FQHQABNCZUHTXMTK3AIHUAGYDQXQYVQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection Dec 9 07:29:49.689861212 (88h0m49.560944247s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection Dec 9 07:29:49.689914644 (88h0m49.560997689s) [noise=true , address=145.79.227.28:4725 , piece_id=AGDXVUHDFM7DC2HNW3YUKREAROODLZBIVUMTFYBY3CXYQ7HSYGWA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection Dec 9 07:29:49.690003323 (88h0m49.561086368s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection Dec 9 07:29:49.690090949 (88h0m49.561173994s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection Dec 9 07:29:49.690262546 (88h0m49.561345591s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection Dec 9 07:29:49.690582726 (88h0m49.561665772s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:50466->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:29:49.690631769 (88h0m49.561714805s) [seg_pos={0 0}, upload=76810]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Dec 9 07:29:50.737500571 (88h0m50.608583616s) [upload=76810]: Done uploading segments... Dec 9 07:29:50.737550175 (88h0m50.60863321s) [upload=76810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 17.009138796s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.926881023s trace: https://jaeger.dev.storj.io/trace/55634d9c1d55ff67 FAIL (2025-12-09 07:43:52 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.926937282s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.736905399s trace: https://jaeger.dev.storj.io/trace/219dd7469694ce8d FAIL (2025-12-09 07:43:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.221678783 (87h58m24.092761818s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222184665 (87h58m24.09326771s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222790810 (87h58m24.093873855s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223285452 (87h58m24.094368487s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223348001 (87h58m24.094431036s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223906905 (87h58m24.09498994s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224273312 (87h58m24.095356357s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224668005 (87h58m24.09575104s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225016438 (87h58m24.096099483s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225398876 (87h58m24.096481911s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225880544 (87h58m24.096963579s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.226416704 (87h58m24.097499739s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.227984600 (87h58m24.099067635s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.229456091 (87h58m24.100539136s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.231011924 (87h58m24.102094969s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.232660333 (87h58m24.103743388s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:25.694926842 (87h58m25.566009888s) [upload=76783]: Next returned nil segment Dec 9 07:27:25.694949827 (87h58m25.566032862s) [upload=76783]: Waiting for error group managing segments... Dec 9 07:27:26.134631394 (87h58m26.00571443s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.144638911 (87h58m26.015721956s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.194703193 (87h58m26.065786228s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.202448432 (87h58m26.073531477s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.218793957 (87h58m26.089876992s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.219708348 (87h58m26.090791393s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.227913963 (87h58m26.098996998s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.238868943 (87h58m26.109951988s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.249699476 (87h58m26.120782522s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.263815877 (87h58m26.134898912s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.264418876 (87h58m26.135501911s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.280320365 (87h58m26.1514034s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.294136013 (87h58m26.165219058s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321886986 (87h58m26.192970031s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321911793 (87h58m26.192994838s) [seg_pos={0 0}, upload=76783]: Segment reached optimal threshold of 14 pieces. Dec 9 07:27:26.322167190 (87h58m26.193250235s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322175145 (87h58m26.19325821s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.322770298 (87h58m26.193853333s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322779535 (87h58m26.19386258s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323079737 (87h58m26.194162792s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT): protocol: read tcp 188.166.225.170:43106->145.79.227.36:4548: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323089335 (87h58m26.19417238s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323372955 (87h58m26.19445599s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: read tcp 188.166.225.170:50502->64.49.13.169:4716: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323381462 (87h58m26.194464507s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323400197 (87h58m26.194483242s) [seg_pos={0 0}, upload=76783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:27:26.323490970 (87h58m26.194574015s) [upload=76783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:27:26.323497242 (87h58m26.194580287s) [upload=76783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:30:43.052543346 (88h1m42.923626391s) [upload=76783]: Done uploading segments... Dec 9 07:30:43.052668744 (88h1m42.923751799s) [upload=76783]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.736956476s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [....................................XXX.....................] duration: 2.160839908s trace: https://jaeger.dev.storj.io/trace/178b4cba623ae119 FAIL (2025-12-09 07:43:50 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.160870878s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X...................X..........................] duration: 5.738228338s trace: https://jaeger.dev.storj.io/trace/59c049b2e292a646 FAIL (2025-12-09 07:43:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.738256493s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...........................X...........XX..X.......X...X....] duration: 5.99156947s trace: https://jaeger.dev.storj.io/trace/76e27d463b59cd34 FAIL (2025-12-09 07:43:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.991666615s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.74250298s trace: https://jaeger.dev.storj.io/trace/219dd7469694ce8d FAIL (2025-12-09 07:43:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.221678783 (87h58m24.092761818s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222184665 (87h58m24.09326771s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222790810 (87h58m24.093873855s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223285452 (87h58m24.094368487s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223348001 (87h58m24.094431036s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223906905 (87h58m24.09498994s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224273312 (87h58m24.095356357s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224668005 (87h58m24.09575104s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225016438 (87h58m24.096099483s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225398876 (87h58m24.096481911s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225880544 (87h58m24.096963579s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.226416704 (87h58m24.097499739s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.227984600 (87h58m24.099067635s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.229456091 (87h58m24.100539136s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.231011924 (87h58m24.102094969s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.232660333 (87h58m24.103743388s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:25.694926842 (87h58m25.566009888s) [upload=76783]: Next returned nil segment Dec 9 07:27:25.694949827 (87h58m25.566032862s) [upload=76783]: Waiting for error group managing segments... Dec 9 07:27:26.134631394 (87h58m26.00571443s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.144638911 (87h58m26.015721956s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.194703193 (87h58m26.065786228s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.202448432 (87h58m26.073531477s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.218793957 (87h58m26.089876992s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.219708348 (87h58m26.090791393s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.227913963 (87h58m26.098996998s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.238868943 (87h58m26.109951988s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.249699476 (87h58m26.120782522s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.263815877 (87h58m26.134898912s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.264418876 (87h58m26.135501911s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.280320365 (87h58m26.1514034s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.294136013 (87h58m26.165219058s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321886986 (87h58m26.192970031s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321911793 (87h58m26.192994838s) [seg_pos={0 0}, upload=76783]: Segment reached optimal threshold of 14 pieces. Dec 9 07:27:26.322167190 (87h58m26.193250235s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322175145 (87h58m26.19325821s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.322770298 (87h58m26.193853333s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322779535 (87h58m26.19386258s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323079737 (87h58m26.194162792s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT): protocol: read tcp 188.166.225.170:43106->145.79.227.36:4548: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323089335 (87h58m26.19417238s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323372955 (87h58m26.19445599s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: read tcp 188.166.225.170:50502->64.49.13.169:4716: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323381462 (87h58m26.194464507s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323400197 (87h58m26.194483242s) [seg_pos={0 0}, upload=76783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:27:26.323490970 (87h58m26.194574015s) [upload=76783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:27:26.323497242 (87h58m26.194580287s) [upload=76783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:30:43.052543346 (88h1m42.923626391s) [upload=76783]: Done uploading segments... Dec 9 07:30:43.052668744 (88h1m42.923751799s) [upload=76783]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.742601185s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.982562252s trace: https://jaeger.dev.storj.io/trace/42dcab7a5e240fd8 FAIL (2025-12-09 07:43:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:28:09.412200246 (87h59m9.283283281s) [upload=76792]: Starting upload Dec 9 07:28:09.412257335 (87h59m9.283340371s) [upload=76792]: Done starting upload Dec 9 07:28:09.412337338 (87h59m9.283420383s) [upload=76792]: Uploading segments... Dec 9 07:28:09.412384597 (87h59m9.283467632s) [upload=76792]: Got next segment. Inline: true Dec 9 07:28:09.412398334 (87h59m9.283481389s) [upload=76792]: Waiting for error group managing segments... Dec 9 07:28:09.412446225 (87h59m9.28352927s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:28:09.412452456 (87h59m9.283535491s) [upload=76792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:28:09.412455432 (87h59m9.283538477s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:28:19.867794892 (87h59m19.738877927s) [upload=76792]: Done uploading segments... Dec 9 07:28:19.867843895 (87h59m19.73892694s) [upload=76792]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.982590496s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.................................XXX.X...................] duration: 5.681073056s trace: https://jaeger.dev.storj.io/trace/34e772a771ff36b2 FAIL (2025-12-09 07:43:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :29:22.029442363 (88h0m21.900525398s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.029471198 (88h0m21.900554233s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.030555563 (88h0m21.901638608s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031156617 (88h0m21.902239662s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031594381 (88h0m21.902677416s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032034670 (88h0m21.903117715s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032167963 (88h0m21.903250998s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032800087 (88h0m21.903883132s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:23.836496806 (88h0m23.707579841s) [upload=76810]: Next returned nil segment Dec 9 07:29:23.836512486 (88h0m23.707595531s) [upload=76810]: Waiting for error group managing segments... Dec 9 07:29:24.124325638 (88h0m23.995408683s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.114522896 (88h0m24.985605941s) [noise=true , address=145.79.227.21:4475 , piece_id=ED4VS3YEIFPJGGKW36BJKVLT3EXGVOFG3SD23LEPPFIAOOKHUD4Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.377083973 (88h0m25.248167008s) [noise=true , address=145.79.227.41:4086 , piece_id=EZUBNMEUWTNJBXJBWJYXFJQEPFFVQOUHZAWVJW6NCWKGRKAS2PSA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.480466423 (88h0m25.351549468s) [noise=true , address=145.79.227.42:4121 , piece_id=7I2TXNXZM2KM5TEDDF7Q5DDUSBAPBE3WWFTTJFGNM5JVVCTQNU5Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:26.767037853 (88h0m26.638120908s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:49.689337786 (88h0m49.560420841s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection Dec 9 07:29:49.689500304 (88h0m49.56058335s) [noise=true , address=145.79.227.34:4907 , piece_id=UG6M3T655YGW4OGBYEVOLQVR7U5XWXPDDNXBZAPTHYP2DYWDRRYA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection Dec 9 07:29:49.689541303 (88h0m49.560624348s) [noise=true , address=145.79.227.33:4870 , piece_id=6Q5XSDCJCDRWHPO7JWCDNPQM5JN2ESDCRXWY5V23AGA6Q4C5Y7HQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection Dec 9 07:29:49.689654439 (88h0m49.560737474s) [noise=true , address=145.79.227.29:4943 , piece_id=FQ2FNNQR6QBWTJOIQBIOU7BSWIA6ZZMDWC4DE42NUNR5Z7ZDGCGQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection Dec 9 07:29:49.689700015 (88h0m49.56078306s) [noise=true , address=145.79.227.42:4122 , piece_id=5WKTAWBKWCZVU3YX4RSF4FQHQABNCZUHTXMTK3AIHUAGYDQXQYVQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection Dec 9 07:29:49.689861212 (88h0m49.560944247s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection Dec 9 07:29:49.689914644 (88h0m49.560997689s) [noise=true , address=145.79.227.28:4725 , piece_id=AGDXVUHDFM7DC2HNW3YUKREAROODLZBIVUMTFYBY3CXYQ7HSYGWA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection Dec 9 07:29:49.690003323 (88h0m49.561086368s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection Dec 9 07:29:49.690090949 (88h0m49.561173994s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection Dec 9 07:29:49.690262546 (88h0m49.561345591s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection Dec 9 07:29:49.690582726 (88h0m49.561665772s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:50466->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:29:49.690631769 (88h0m49.561714805s) [seg_pos={0 0}, upload=76810]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Dec 9 07:29:50.737500571 (88h0m50.608583616s) [upload=76810]: Done uploading segments... Dec 9 07:29:50.737550175 (88h0m50.60863321s) [upload=76810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.681094416s core uplink us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 433.560895ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt trace: https://jaeger.dev.storj.io/trace/51a280a8635ce0b9 FAIL (2025-12-09 07:43:22 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 433.586433ms edge linksharing us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......................................X.XX.XX..X............] duration: 3.441931918s trace: https://jaeger.dev.storj.io/trace/402bfb4648c7d69b FAIL (2025-12-09 07:43:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.441961814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.78418606s trace: https://jaeger.dev.storj.io/trace/219dd7469694ce8d FAIL (2025-12-09 07:43:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.221678783 (87h58m24.092761818s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222184665 (87h58m24.09326771s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222790810 (87h58m24.093873855s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223285452 (87h58m24.094368487s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223348001 (87h58m24.094431036s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223906905 (87h58m24.09498994s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224273312 (87h58m24.095356357s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224668005 (87h58m24.09575104s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225016438 (87h58m24.096099483s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225398876 (87h58m24.096481911s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225880544 (87h58m24.096963579s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.226416704 (87h58m24.097499739s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.227984600 (87h58m24.099067635s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.229456091 (87h58m24.100539136s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.231011924 (87h58m24.102094969s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.232660333 (87h58m24.103743388s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:25.694926842 (87h58m25.566009888s) [upload=76783]: Next returned nil segment Dec 9 07:27:25.694949827 (87h58m25.566032862s) [upload=76783]: Waiting for error group managing segments... Dec 9 07:27:26.134631394 (87h58m26.00571443s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.144638911 (87h58m26.015721956s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.194703193 (87h58m26.065786228s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.202448432 (87h58m26.073531477s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.218793957 (87h58m26.089876992s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.219708348 (87h58m26.090791393s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.227913963 (87h58m26.098996998s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.238868943 (87h58m26.109951988s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.249699476 (87h58m26.120782522s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.263815877 (87h58m26.134898912s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.264418876 (87h58m26.135501911s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.280320365 (87h58m26.1514034s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.294136013 (87h58m26.165219058s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321886986 (87h58m26.192970031s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321911793 (87h58m26.192994838s) [seg_pos={0 0}, upload=76783]: Segment reached optimal threshold of 14 pieces. Dec 9 07:27:26.322167190 (87h58m26.193250235s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322175145 (87h58m26.19325821s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.322770298 (87h58m26.193853333s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322779535 (87h58m26.19386258s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323079737 (87h58m26.194162792s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT): protocol: read tcp 188.166.225.170:43106->145.79.227.36:4548: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323089335 (87h58m26.19417238s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323372955 (87h58m26.19445599s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: read tcp 188.166.225.170:50502->64.49.13.169:4716: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323381462 (87h58m26.194464507s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323400197 (87h58m26.194483242s) [seg_pos={0 0}, upload=76783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:27:26.323490970 (87h58m26.194574015s) [upload=76783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:27:26.323497242 (87h58m26.194580287s) [upload=76783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:30:43.052543346 (88h1m42.923626391s) [upload=76783]: Done uploading segments... Dec 9 07:30:43.052668744 (88h1m42.923751799s) [upload=76783]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.784215408s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....................................XXX....................] duration: 2.203036979s trace: https://jaeger.dev.storj.io/trace/178b4cba623ae119 FAIL (2025-12-09 07:43:07 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.203067377s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 446.983964ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt trace: https://jaeger.dev.storj.io/trace/51a280a8635ce0b9 FAIL (2025-12-09 07:43:05 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 447.013892ms edge linksharing us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 5.996506162s trace: https://jaeger.dev.storj.io/trace/5f10a1154ce06783 FAIL (2025-12-09 07:43:05 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.996551858s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 4.980711902s trace: https://jaeger.dev.storj.io/trace/50f433cc1e60731c FAIL (2025-12-09 07:43:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.980739494s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X...................X.........................] duration: 5.832946493s trace: https://jaeger.dev.storj.io/trace/59c049b2e292a646 FAIL (2025-12-09 07:43:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.832978564s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 1.957455266s trace: https://jaeger.dev.storj.io/trace/55634d9c1d55ff67 FAIL (2025-12-09 07:42:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.957492437s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.986004027s trace: https://jaeger.dev.storj.io/trace/42dcab7a5e240fd8 FAIL (2025-12-09 07:42:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:28:09.412200246 (87h59m9.283283281s) [upload=76792]: Starting upload Dec 9 07:28:09.412257335 (87h59m9.283340371s) [upload=76792]: Done starting upload Dec 9 07:28:09.412337338 (87h59m9.283420383s) [upload=76792]: Uploading segments... Dec 9 07:28:09.412384597 (87h59m9.283467632s) [upload=76792]: Got next segment. Inline: true Dec 9 07:28:09.412398334 (87h59m9.283481389s) [upload=76792]: Waiting for error group managing segments... Dec 9 07:28:09.412446225 (87h59m9.28352927s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:28:09.412452456 (87h59m9.283535491s) [upload=76792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:28:09.412455432 (87h59m9.283538477s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:28:19.867794892 (87h59m19.738877927s) [upload=76792]: Done uploading segments... Dec 9 07:28:19.867843895 (87h59m19.73892694s) [upload=76792]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.986033784s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.................................XXX.X..................] duration: 5.676001961s trace: https://jaeger.dev.storj.io/trace/34e772a771ff36b2 FAIL (2025-12-09 07:42:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :29:22.029442363 (88h0m21.900525398s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.029471198 (88h0m21.900554233s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.030555563 (88h0m21.901638608s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031156617 (88h0m21.902239662s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031594381 (88h0m21.902677416s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032034670 (88h0m21.903117715s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032167963 (88h0m21.903250998s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032800087 (88h0m21.903883132s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:23.836496806 (88h0m23.707579841s) [upload=76810]: Next returned nil segment Dec 9 07:29:23.836512486 (88h0m23.707595531s) [upload=76810]: Waiting for error group managing segments... Dec 9 07:29:24.124325638 (88h0m23.995408683s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.114522896 (88h0m24.985605941s) [noise=true , address=145.79.227.21:4475 , piece_id=ED4VS3YEIFPJGGKW36BJKVLT3EXGVOFG3SD23LEPPFIAOOKHUD4Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.377083973 (88h0m25.248167008s) [noise=true , address=145.79.227.41:4086 , piece_id=EZUBNMEUWTNJBXJBWJYXFJQEPFFVQOUHZAWVJW6NCWKGRKAS2PSA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.480466423 (88h0m25.351549468s) [noise=true , address=145.79.227.42:4121 , piece_id=7I2TXNXZM2KM5TEDDF7Q5DDUSBAPBE3WWFTTJFGNM5JVVCTQNU5Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:26.767037853 (88h0m26.638120908s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:49.689337786 (88h0m49.560420841s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection Dec 9 07:29:49.689500304 (88h0m49.56058335s) [noise=true , address=145.79.227.34:4907 , piece_id=UG6M3T655YGW4OGBYEVOLQVR7U5XWXPDDNXBZAPTHYP2DYWDRRYA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection Dec 9 07:29:49.689541303 (88h0m49.560624348s) [noise=true , address=145.79.227.33:4870 , piece_id=6Q5XSDCJCDRWHPO7JWCDNPQM5JN2ESDCRXWY5V23AGA6Q4C5Y7HQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection Dec 9 07:29:49.689654439 (88h0m49.560737474s) [noise=true , address=145.79.227.29:4943 , piece_id=FQ2FNNQR6QBWTJOIQBIOU7BSWIA6ZZMDWC4DE42NUNR5Z7ZDGCGQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection Dec 9 07:29:49.689700015 (88h0m49.56078306s) [noise=true , address=145.79.227.42:4122 , piece_id=5WKTAWBKWCZVU3YX4RSF4FQHQABNCZUHTXMTK3AIHUAGYDQXQYVQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection Dec 9 07:29:49.689861212 (88h0m49.560944247s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection Dec 9 07:29:49.689914644 (88h0m49.560997689s) [noise=true , address=145.79.227.28:4725 , piece_id=AGDXVUHDFM7DC2HNW3YUKREAROODLZBIVUMTFYBY3CXYQ7HSYGWA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection Dec 9 07:29:49.690003323 (88h0m49.561086368s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection Dec 9 07:29:49.690090949 (88h0m49.561173994s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection Dec 9 07:29:49.690262546 (88h0m49.561345591s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection Dec 9 07:29:49.690582726 (88h0m49.561665772s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:50466->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:29:49.690631769 (88h0m49.561714805s) [seg_pos={0 0}, upload=76810]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Dec 9 07:29:50.737500571 (88h0m50.608583616s) [upload=76810]: Done uploading segments... Dec 9 07:29:50.737550175 (88h0m50.60863321s) [upload=76810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.676029823s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 3 out of 60 is failed (threshold=3) [......................................XX..X.................] duration: 698.257608ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/43834d9fac36e837 FAIL (2025-12-09 07:42:50 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 698.284278ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 6.067161904s trace: https://jaeger.dev.storj.io/trace/5f10a1154ce06783 FAIL (2025-12-09 07:42:48 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.067185699s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [............................X...........XX..X.......X...X...] duration: 5.967809446s trace: https://jaeger.dev.storj.io/trace/76e27d463b59cd34 FAIL (2025-12-09 07:42:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.967833423s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.......................................X.XX.XX..X...........] duration: 3.429432736s trace: https://jaeger.dev.storj.io/trace/402bfb4648c7d69b FAIL (2025-12-09 07:42:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.429459065s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [........................................X.XX.XX..X..........] duration: 3.581318525s trace: https://jaeger.dev.storj.io/trace/402bfb4648c7d69b FAIL (2025-12-09 07:42:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.581356848s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [X............................X...........XX..X.......X...X..] duration: 6.304003175s trace: https://jaeger.dev.storj.io/trace/76e27d463b59cd34 FAIL (2025-12-09 07:42:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.304032082s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.987119719s trace: https://jaeger.dev.storj.io/trace/42dcab7a5e240fd8 FAIL (2025-12-09 07:42:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:28:09.412200246 (87h59m9.283283281s) [upload=76792]: Starting upload Dec 9 07:28:09.412257335 (87h59m9.283340371s) [upload=76792]: Done starting upload Dec 9 07:28:09.412337338 (87h59m9.283420383s) [upload=76792]: Uploading segments... Dec 9 07:28:09.412384597 (87h59m9.283467632s) [upload=76792]: Got next segment. Inline: true Dec 9 07:28:09.412398334 (87h59m9.283481389s) [upload=76792]: Waiting for error group managing segments... Dec 9 07:28:09.412446225 (87h59m9.28352927s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:28:09.412452456 (87h59m9.283535491s) [upload=76792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:28:09.412455432 (87h59m9.283538477s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:28:19.867794892 (87h59m19.738877927s) [upload=76792]: Done uploading segments... Dec 9 07:28:19.867843895 (87h59m19.73892694s) [upload=76792]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.98714658s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [......................................XXX...................] duration: 2.15144886s trace: https://jaeger.dev.storj.io/trace/178b4cba623ae119 FAIL (2025-12-09 07:42:08 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.151548039s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.971971469s trace: https://jaeger.dev.storj.io/trace/55634d9c1d55ff67 FAIL (2025-12-09 07:42:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.972002047s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.069643714s trace: https://jaeger.dev.storj.io/trace/219dd7469694ce8d FAIL (2025-12-09 07:42:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.221678783 (87h58m24.092761818s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222184665 (87h58m24.09326771s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222790810 (87h58m24.093873855s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223285452 (87h58m24.094368487s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223348001 (87h58m24.094431036s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223906905 (87h58m24.09498994s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224273312 (87h58m24.095356357s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224668005 (87h58m24.09575104s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225016438 (87h58m24.096099483s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225398876 (87h58m24.096481911s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225880544 (87h58m24.096963579s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.226416704 (87h58m24.097499739s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.227984600 (87h58m24.099067635s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.229456091 (87h58m24.100539136s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.231011924 (87h58m24.102094969s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.232660333 (87h58m24.103743388s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:25.694926842 (87h58m25.566009888s) [upload=76783]: Next returned nil segment Dec 9 07:27:25.694949827 (87h58m25.566032862s) [upload=76783]: Waiting for error group managing segments... Dec 9 07:27:26.134631394 (87h58m26.00571443s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.144638911 (87h58m26.015721956s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.194703193 (87h58m26.065786228s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.202448432 (87h58m26.073531477s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.218793957 (87h58m26.089876992s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.219708348 (87h58m26.090791393s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.227913963 (87h58m26.098996998s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.238868943 (87h58m26.109951988s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.249699476 (87h58m26.120782522s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.263815877 (87h58m26.134898912s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.264418876 (87h58m26.135501911s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.280320365 (87h58m26.1514034s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.294136013 (87h58m26.165219058s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321886986 (87h58m26.192970031s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321911793 (87h58m26.192994838s) [seg_pos={0 0}, upload=76783]: Segment reached optimal threshold of 14 pieces. Dec 9 07:27:26.322167190 (87h58m26.193250235s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322175145 (87h58m26.19325821s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.322770298 (87h58m26.193853333s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322779535 (87h58m26.19386258s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323079737 (87h58m26.194162792s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT): protocol: read tcp 188.166.225.170:43106->145.79.227.36:4548: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323089335 (87h58m26.19417238s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323372955 (87h58m26.19445599s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: read tcp 188.166.225.170:50502->64.49.13.169:4716: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323381462 (87h58m26.194464507s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323400197 (87h58m26.194483242s) [seg_pos={0 0}, upload=76783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:27:26.323490970 (87h58m26.194574015s) [upload=76783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:27:26.323497242 (87h58m26.194580287s) [upload=76783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:30:43.052543346 (88h1m42.923626391s) [upload=76783]: Done uploading segments... Dec 9 07:30:43.052668744 (88h1m42.923751799s) [upload=76783]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.069671427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.................................XXX.X.................] duration: 5.577005499s trace: https://jaeger.dev.storj.io/trace/34e772a771ff36b2 FAIL (2025-12-09 07:42:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :29:22.029442363 (88h0m21.900525398s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.029471198 (88h0m21.900554233s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.030555563 (88h0m21.901638608s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031156617 (88h0m21.902239662s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031594381 (88h0m21.902677416s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032034670 (88h0m21.903117715s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032167963 (88h0m21.903250998s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032800087 (88h0m21.903883132s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:23.836496806 (88h0m23.707579841s) [upload=76810]: Next returned nil segment Dec 9 07:29:23.836512486 (88h0m23.707595531s) [upload=76810]: Waiting for error group managing segments... Dec 9 07:29:24.124325638 (88h0m23.995408683s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.114522896 (88h0m24.985605941s) [noise=true , address=145.79.227.21:4475 , piece_id=ED4VS3YEIFPJGGKW36BJKVLT3EXGVOFG3SD23LEPPFIAOOKHUD4Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.377083973 (88h0m25.248167008s) [noise=true , address=145.79.227.41:4086 , piece_id=EZUBNMEUWTNJBXJBWJYXFJQEPFFVQOUHZAWVJW6NCWKGRKAS2PSA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.480466423 (88h0m25.351549468s) [noise=true , address=145.79.227.42:4121 , piece_id=7I2TXNXZM2KM5TEDDF7Q5DDUSBAPBE3WWFTTJFGNM5JVVCTQNU5Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:26.767037853 (88h0m26.638120908s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:49.689337786 (88h0m49.560420841s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection Dec 9 07:29:49.689500304 (88h0m49.56058335s) [noise=true , address=145.79.227.34:4907 , piece_id=UG6M3T655YGW4OGBYEVOLQVR7U5XWXPDDNXBZAPTHYP2DYWDRRYA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection Dec 9 07:29:49.689541303 (88h0m49.560624348s) [noise=true , address=145.79.227.33:4870 , piece_id=6Q5XSDCJCDRWHPO7JWCDNPQM5JN2ESDCRXWY5V23AGA6Q4C5Y7HQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection Dec 9 07:29:49.689654439 (88h0m49.560737474s) [noise=true , address=145.79.227.29:4943 , piece_id=FQ2FNNQR6QBWTJOIQBIOU7BSWIA6ZZMDWC4DE42NUNR5Z7ZDGCGQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection Dec 9 07:29:49.689700015 (88h0m49.56078306s) [noise=true , address=145.79.227.42:4122 , piece_id=5WKTAWBKWCZVU3YX4RSF4FQHQABNCZUHTXMTK3AIHUAGYDQXQYVQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection Dec 9 07:29:49.689861212 (88h0m49.560944247s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection Dec 9 07:29:49.689914644 (88h0m49.560997689s) [noise=true , address=145.79.227.28:4725 , piece_id=AGDXVUHDFM7DC2HNW3YUKREAROODLZBIVUMTFYBY3CXYQ7HSYGWA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection Dec 9 07:29:49.690003323 (88h0m49.561086368s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection Dec 9 07:29:49.690090949 (88h0m49.561173994s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection Dec 9 07:29:49.690262546 (88h0m49.561345591s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection Dec 9 07:29:49.690582726 (88h0m49.561665772s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:50466->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:29:49.690631769 (88h0m49.561714805s) [seg_pos={0 0}, upload=76810]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Dec 9 07:29:50.737500571 (88h0m50.608583616s) [upload=76810]: Done uploading segments... Dec 9 07:29:50.737550175 (88h0m50.60863321s) [upload=76810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.577041588s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X...................X........................] duration: 5.772422514s trace: https://jaeger.dev.storj.io/trace/59c049b2e292a646 FAIL (2025-12-09 07:42:01 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.772451949s core uplink us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 4.973717241s trace: https://jaeger.dev.storj.io/trace/50f433cc1e60731c FAIL (2025-12-09 07:41:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.973744753s core uplink eu1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 438.029734ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt trace: https://jaeger.dev.storj.io/trace/51a280a8635ce0b9 FAIL (2025-12-09 07:41:54 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 438.057095ms edge linksharing us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 5.732785233s trace: https://jaeger.dev.storj.io/trace/5f10a1154ce06783 FAIL (2025-12-09 07:41:50 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.732860436s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.X............................X...........XX..X.......X...X.] duration: 5.694139353s trace: https://jaeger.dev.storj.io/trace/76e27d463b59cd34 FAIL (2025-12-09 07:41:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.694161245s core uplink us-select-1 us1-placement-20

1
...
11628
11629
11630
11631
11632
...
17366