Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.............................................X.XX.XX..X.....] duration: 3.487489235s trace: https://jaeger.dev.storj.io/trace/402bfb4648c7d69b FAIL (2025-12-09 07:37:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.487558357s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.904857266s trace: https://jaeger.dev.storj.io/trace/55634d9c1d55ff67 FAIL (2025-12-09 07:36:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.904884609s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................XX.............] duration: 6.135759254s trace: https://jaeger.dev.storj.io/trace/5f10a1154ce06783 FAIL (2025-12-09 07:36:55 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.135790043s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.706399169s trace: https://jaeger.dev.storj.io/trace/219dd7469694ce8d FAIL (2025-12-09 07:36:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.221678783 (87h58m24.092761818s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222184665 (87h58m24.09326771s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222790810 (87h58m24.093873855s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223285452 (87h58m24.094368487s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223348001 (87h58m24.094431036s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223906905 (87h58m24.09498994s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224273312 (87h58m24.095356357s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224668005 (87h58m24.09575104s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225016438 (87h58m24.096099483s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225398876 (87h58m24.096481911s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225880544 (87h58m24.096963579s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.226416704 (87h58m24.097499739s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.227984600 (87h58m24.099067635s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.229456091 (87h58m24.100539136s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.231011924 (87h58m24.102094969s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.232660333 (87h58m24.103743388s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:25.694926842 (87h58m25.566009888s) [upload=76783]: Next returned nil segment Dec 9 07:27:25.694949827 (87h58m25.566032862s) [upload=76783]: Waiting for error group managing segments... Dec 9 07:27:26.134631394 (87h58m26.00571443s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.144638911 (87h58m26.015721956s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.194703193 (87h58m26.065786228s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.202448432 (87h58m26.073531477s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.218793957 (87h58m26.089876992s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.219708348 (87h58m26.090791393s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.227913963 (87h58m26.098996998s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.238868943 (87h58m26.109951988s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.249699476 (87h58m26.120782522s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.263815877 (87h58m26.134898912s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.264418876 (87h58m26.135501911s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.280320365 (87h58m26.1514034s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.294136013 (87h58m26.165219058s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321886986 (87h58m26.192970031s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321911793 (87h58m26.192994838s) [seg_pos={0 0}, upload=76783]: Segment reached optimal threshold of 14 pieces. Dec 9 07:27:26.322167190 (87h58m26.193250235s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322175145 (87h58m26.19325821s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.322770298 (87h58m26.193853333s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322779535 (87h58m26.19386258s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323079737 (87h58m26.194162792s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT): protocol: read tcp 188.166.225.170:43106->145.79.227.36:4548: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323089335 (87h58m26.19417238s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323372955 (87h58m26.19445599s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: read tcp 188.166.225.170:50502->64.49.13.169:4716: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323381462 (87h58m26.194464507s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323400197 (87h58m26.194483242s) [seg_pos={0 0}, upload=76783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:27:26.323490970 (87h58m26.194574015s) [upload=76783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:27:26.323497242 (87h58m26.194580287s) [upload=76783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:30:43.052543346 (88h1m42.923626391s) [upload=76783]: Done uploading segments... Dec 9 07:30:43.052668744 (88h1m42.923751799s) [upload=76783]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.706438324s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 3 out of 60 is failed (threshold=3) [..............................................XX..X.........] duration: 706.96601ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/43834d9fac36e837 FAIL (2025-12-09 07:36:41 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 707.093542ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X.................................XXX.X..........] duration: 5.679223152s trace: https://jaeger.dev.storj.io/trace/34e772a771ff36b2 FAIL (2025-12-09 07:36:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :29:22.029442363 (88h0m21.900525398s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.029471198 (88h0m21.900554233s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.030555563 (88h0m21.901638608s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031156617 (88h0m21.902239662s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031594381 (88h0m21.902677416s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032034670 (88h0m21.903117715s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032167963 (88h0m21.903250998s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032800087 (88h0m21.903883132s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:23.836496806 (88h0m23.707579841s) [upload=76810]: Next returned nil segment Dec 9 07:29:23.836512486 (88h0m23.707595531s) [upload=76810]: Waiting for error group managing segments... Dec 9 07:29:24.124325638 (88h0m23.995408683s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.114522896 (88h0m24.985605941s) [noise=true , address=145.79.227.21:4475 , piece_id=ED4VS3YEIFPJGGKW36BJKVLT3EXGVOFG3SD23LEPPFIAOOKHUD4Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.377083973 (88h0m25.248167008s) [noise=true , address=145.79.227.41:4086 , piece_id=EZUBNMEUWTNJBXJBWJYXFJQEPFFVQOUHZAWVJW6NCWKGRKAS2PSA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.480466423 (88h0m25.351549468s) [noise=true , address=145.79.227.42:4121 , piece_id=7I2TXNXZM2KM5TEDDF7Q5DDUSBAPBE3WWFTTJFGNM5JVVCTQNU5Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:26.767037853 (88h0m26.638120908s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:49.689337786 (88h0m49.560420841s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection Dec 9 07:29:49.689500304 (88h0m49.56058335s) [noise=true , address=145.79.227.34:4907 , piece_id=UG6M3T655YGW4OGBYEVOLQVR7U5XWXPDDNXBZAPTHYP2DYWDRRYA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection Dec 9 07:29:49.689541303 (88h0m49.560624348s) [noise=true , address=145.79.227.33:4870 , piece_id=6Q5XSDCJCDRWHPO7JWCDNPQM5JN2ESDCRXWY5V23AGA6Q4C5Y7HQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection Dec 9 07:29:49.689654439 (88h0m49.560737474s) [noise=true , address=145.79.227.29:4943 , piece_id=FQ2FNNQR6QBWTJOIQBIOU7BSWIA6ZZMDWC4DE42NUNR5Z7ZDGCGQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection Dec 9 07:29:49.689700015 (88h0m49.56078306s) [noise=true , address=145.79.227.42:4122 , piece_id=5WKTAWBKWCZVU3YX4RSF4FQHQABNCZUHTXMTK3AIHUAGYDQXQYVQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection Dec 9 07:29:49.689861212 (88h0m49.560944247s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection Dec 9 07:29:49.689914644 (88h0m49.560997689s) [noise=true , address=145.79.227.28:4725 , piece_id=AGDXVUHDFM7DC2HNW3YUKREAROODLZBIVUMTFYBY3CXYQ7HSYGWA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection Dec 9 07:29:49.690003323 (88h0m49.561086368s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection Dec 9 07:29:49.690090949 (88h0m49.561173994s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection Dec 9 07:29:49.690262546 (88h0m49.561345591s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection Dec 9 07:29:49.690582726 (88h0m49.561665772s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:50466->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:29:49.690631769 (88h0m49.561714805s) [seg_pos={0 0}, upload=76810]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Dec 9 07:29:50.737500571 (88h0m50.608583616s) [upload=76810]: Done uploading segments... Dec 9 07:29:50.737550175 (88h0m50.60863321s) [upload=76810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.679251776s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...................X..................] duration: 5.675332152s trace: https://jaeger.dev.storj.io/trace/59c049b2e292a646 FAIL (2025-12-09 07:36:37 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.675361439s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...X...X............................X...........XX..X.......] duration: 5.516681939s trace: https://jaeger.dev.storj.io/trace/27a6ebfbc56e96c0 FAIL (2025-12-09 07:36:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.516706626s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 1.965285152s trace: https://jaeger.dev.storj.io/trace/55634d9c1d55ff67 FAIL (2025-12-09 07:36:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.965312363s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [............................................XXX.............] duration: 2.233481512s trace: https://jaeger.dev.storj.io/trace/178b4cba623ae119 FAIL (2025-12-09 07:36:24 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.233515135s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................XX............] duration: 5.826830934s trace: https://jaeger.dev.storj.io/trace/5f10a1154ce06783 FAIL (2025-12-09 07:36:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.826857606s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.978199484s trace: https://jaeger.dev.storj.io/trace/42dcab7a5e240fd8 FAIL (2025-12-09 07:36:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:28:09.412200246 (87h59m9.283283281s) [upload=76792]: Starting upload Dec 9 07:28:09.412257335 (87h59m9.283340371s) [upload=76792]: Done starting upload Dec 9 07:28:09.412337338 (87h59m9.283420383s) [upload=76792]: Uploading segments... Dec 9 07:28:09.412384597 (87h59m9.283467632s) [upload=76792]: Got next segment. Inline: true Dec 9 07:28:09.412398334 (87h59m9.283481389s) [upload=76792]: Waiting for error group managing segments... Dec 9 07:28:09.412446225 (87h59m9.28352927s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:28:09.412452456 (87h59m9.283535491s) [upload=76792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:28:09.412455432 (87h59m9.283538477s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:28:19.867794892 (87h59m19.738877927s) [upload=76792]: Done uploading segments... Dec 9 07:28:19.867843895 (87h59m19.73892694s) [upload=76792]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.978228409s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X.................................XXX.X.........] duration: 5.854610152s trace: https://jaeger.dev.storj.io/trace/34e772a771ff36b2 FAIL (2025-12-09 07:36:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :29:22.029442363 (88h0m21.900525398s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.029471198 (88h0m21.900554233s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.030555563 (88h0m21.901638608s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031156617 (88h0m21.902239662s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031594381 (88h0m21.902677416s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032034670 (88h0m21.903117715s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032167963 (88h0m21.903250998s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032800087 (88h0m21.903883132s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:23.836496806 (88h0m23.707579841s) [upload=76810]: Next returned nil segment Dec 9 07:29:23.836512486 (88h0m23.707595531s) [upload=76810]: Waiting for error group managing segments... Dec 9 07:29:24.124325638 (88h0m23.995408683s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.114522896 (88h0m24.985605941s) [noise=true , address=145.79.227.21:4475 , piece_id=ED4VS3YEIFPJGGKW36BJKVLT3EXGVOFG3SD23LEPPFIAOOKHUD4Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.377083973 (88h0m25.248167008s) [noise=true , address=145.79.227.41:4086 , piece_id=EZUBNMEUWTNJBXJBWJYXFJQEPFFVQOUHZAWVJW6NCWKGRKAS2PSA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.480466423 (88h0m25.351549468s) [noise=true , address=145.79.227.42:4121 , piece_id=7I2TXNXZM2KM5TEDDF7Q5DDUSBAPBE3WWFTTJFGNM5JVVCTQNU5Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:26.767037853 (88h0m26.638120908s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:49.689337786 (88h0m49.560420841s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection Dec 9 07:29:49.689500304 (88h0m49.56058335s) [noise=true , address=145.79.227.34:4907 , piece_id=UG6M3T655YGW4OGBYEVOLQVR7U5XWXPDDNXBZAPTHYP2DYWDRRYA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection Dec 9 07:29:49.689541303 (88h0m49.560624348s) [noise=true , address=145.79.227.33:4870 , piece_id=6Q5XSDCJCDRWHPO7JWCDNPQM5JN2ESDCRXWY5V23AGA6Q4C5Y7HQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection Dec 9 07:29:49.689654439 (88h0m49.560737474s) [noise=true , address=145.79.227.29:4943 , piece_id=FQ2FNNQR6QBWTJOIQBIOU7BSWIA6ZZMDWC4DE42NUNR5Z7ZDGCGQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection Dec 9 07:29:49.689700015 (88h0m49.56078306s) [noise=true , address=145.79.227.42:4122 , piece_id=5WKTAWBKWCZVU3YX4RSF4FQHQABNCZUHTXMTK3AIHUAGYDQXQYVQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection Dec 9 07:29:49.689861212 (88h0m49.560944247s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection Dec 9 07:29:49.689914644 (88h0m49.560997689s) [noise=true , address=145.79.227.28:4725 , piece_id=AGDXVUHDFM7DC2HNW3YUKREAROODLZBIVUMTFYBY3CXYQ7HSYGWA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection Dec 9 07:29:49.690003323 (88h0m49.561086368s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection Dec 9 07:29:49.690090949 (88h0m49.561173994s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection Dec 9 07:29:49.690262546 (88h0m49.561345591s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection Dec 9 07:29:49.690582726 (88h0m49.561665772s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:50466->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:29:49.690631769 (88h0m49.561714805s) [seg_pos={0 0}, upload=76810]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Dec 9 07:29:50.737500571 (88h0m50.608583616s) [upload=76810]: Done uploading segments... Dec 9 07:29:50.737550175 (88h0m50.60863321s) [upload=76810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.854635461s core uplink us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 435.744622ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt trace: https://jaeger.dev.storj.io/trace/51a280a8635ce0b9 FAIL (2025-12-09 07:36:11 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 435.778066ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 4.884527007s trace: https://jaeger.dev.storj.io/trace/50f433cc1e60731c FAIL (2025-12-09 07:36:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.884557826s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.734431071s trace: https://jaeger.dev.storj.io/trace/219dd7469694ce8d FAIL (2025-12-09 07:36:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.221678783 (87h58m24.092761818s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222184665 (87h58m24.09326771s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222790810 (87h58m24.093873855s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223285452 (87h58m24.094368487s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223348001 (87h58m24.094431036s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223906905 (87h58m24.09498994s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224273312 (87h58m24.095356357s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224668005 (87h58m24.09575104s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225016438 (87h58m24.096099483s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225398876 (87h58m24.096481911s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225880544 (87h58m24.096963579s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.226416704 (87h58m24.097499739s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.227984600 (87h58m24.099067635s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.229456091 (87h58m24.100539136s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.231011924 (87h58m24.102094969s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.232660333 (87h58m24.103743388s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:25.694926842 (87h58m25.566009888s) [upload=76783]: Next returned nil segment Dec 9 07:27:25.694949827 (87h58m25.566032862s) [upload=76783]: Waiting for error group managing segments... Dec 9 07:27:26.134631394 (87h58m26.00571443s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.144638911 (87h58m26.015721956s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.194703193 (87h58m26.065786228s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.202448432 (87h58m26.073531477s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.218793957 (87h58m26.089876992s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.219708348 (87h58m26.090791393s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.227913963 (87h58m26.098996998s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.238868943 (87h58m26.109951988s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.249699476 (87h58m26.120782522s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.263815877 (87h58m26.134898912s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.264418876 (87h58m26.135501911s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.280320365 (87h58m26.1514034s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.294136013 (87h58m26.165219058s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321886986 (87h58m26.192970031s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321911793 (87h58m26.192994838s) [seg_pos={0 0}, upload=76783]: Segment reached optimal threshold of 14 pieces. Dec 9 07:27:26.322167190 (87h58m26.193250235s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322175145 (87h58m26.19325821s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.322770298 (87h58m26.193853333s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322779535 (87h58m26.19386258s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323079737 (87h58m26.194162792s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT): protocol: read tcp 188.166.225.170:43106->145.79.227.36:4548: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323089335 (87h58m26.19417238s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323372955 (87h58m26.19445599s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: read tcp 188.166.225.170:50502->64.49.13.169:4716: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323381462 (87h58m26.194464507s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323400197 (87h58m26.194483242s) [seg_pos={0 0}, upload=76783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:27:26.323490970 (87h58m26.194574015s) [upload=76783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:27:26.323497242 (87h58m26.194580287s) [upload=76783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:30:43.052543346 (88h1m42.923626391s) [upload=76783]: Done uploading segments... Dec 9 07:30:43.052668744 (88h1m42.923751799s) [upload=76783]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.734468191s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 1.94308986s trace: https://jaeger.dev.storj.io/trace/55634d9c1d55ff67 FAIL (2025-12-09 07:36:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.943118965s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [X...X...X............................X...........XX..X......] duration: 5.87335093s trace: https://jaeger.dev.storj.io/trace/27a6ebfbc56e96c0 FAIL (2025-12-09 07:36:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.873556971s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................XX...........] duration: 6.221715863s trace: https://jaeger.dev.storj.io/trace/5f10a1154ce06783 FAIL (2025-12-09 07:35:58 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.221741844s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..............................................X.XX.XX..X....] duration: 3.373215836s trace: https://jaeger.dev.storj.io/trace/402bfb4648c7d69b FAIL (2025-12-09 07:35:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.373246335s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X.................................XXX.X........] duration: 5.655913836s trace: https://jaeger.dev.storj.io/trace/34e772a771ff36b2 FAIL (2025-12-09 07:35:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :29:22.029442363 (88h0m21.900525398s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.029471198 (88h0m21.900554233s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.030555563 (88h0m21.901638608s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031156617 (88h0m21.902239662s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031594381 (88h0m21.902677416s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032034670 (88h0m21.903117715s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032167963 (88h0m21.903250998s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032800087 (88h0m21.903883132s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:23.836496806 (88h0m23.707579841s) [upload=76810]: Next returned nil segment Dec 9 07:29:23.836512486 (88h0m23.707595531s) [upload=76810]: Waiting for error group managing segments... Dec 9 07:29:24.124325638 (88h0m23.995408683s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.114522896 (88h0m24.985605941s) [noise=true , address=145.79.227.21:4475 , piece_id=ED4VS3YEIFPJGGKW36BJKVLT3EXGVOFG3SD23LEPPFIAOOKHUD4Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.377083973 (88h0m25.248167008s) [noise=true , address=145.79.227.41:4086 , piece_id=EZUBNMEUWTNJBXJBWJYXFJQEPFFVQOUHZAWVJW6NCWKGRKAS2PSA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.480466423 (88h0m25.351549468s) [noise=true , address=145.79.227.42:4121 , piece_id=7I2TXNXZM2KM5TEDDF7Q5DDUSBAPBE3WWFTTJFGNM5JVVCTQNU5Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:26.767037853 (88h0m26.638120908s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:49.689337786 (88h0m49.560420841s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection Dec 9 07:29:49.689500304 (88h0m49.56058335s) [noise=true , address=145.79.227.34:4907 , piece_id=UG6M3T655YGW4OGBYEVOLQVR7U5XWXPDDNXBZAPTHYP2DYWDRRYA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection Dec 9 07:29:49.689541303 (88h0m49.560624348s) [noise=true , address=145.79.227.33:4870 , piece_id=6Q5XSDCJCDRWHPO7JWCDNPQM5JN2ESDCRXWY5V23AGA6Q4C5Y7HQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection Dec 9 07:29:49.689654439 (88h0m49.560737474s) [noise=true , address=145.79.227.29:4943 , piece_id=FQ2FNNQR6QBWTJOIQBIOU7BSWIA6ZZMDWC4DE42NUNR5Z7ZDGCGQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection Dec 9 07:29:49.689700015 (88h0m49.56078306s) [noise=true , address=145.79.227.42:4122 , piece_id=5WKTAWBKWCZVU3YX4RSF4FQHQABNCZUHTXMTK3AIHUAGYDQXQYVQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection Dec 9 07:29:49.689861212 (88h0m49.560944247s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection Dec 9 07:29:49.689914644 (88h0m49.560997689s) [noise=true , address=145.79.227.28:4725 , piece_id=AGDXVUHDFM7DC2HNW3YUKREAROODLZBIVUMTFYBY3CXYQ7HSYGWA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection Dec 9 07:29:49.690003323 (88h0m49.561086368s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection Dec 9 07:29:49.690090949 (88h0m49.561173994s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection Dec 9 07:29:49.690262546 (88h0m49.561345591s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection Dec 9 07:29:49.690582726 (88h0m49.561665772s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:50466->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:29:49.690631769 (88h0m49.561714805s) [seg_pos={0 0}, upload=76810]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Dec 9 07:29:50.737500571 (88h0m50.608583616s) [upload=76810]: Done uploading segments... Dec 9 07:29:50.737550175 (88h0m50.60863321s) [upload=76810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.655944854s core uplink us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 3 out of 60 is failed (threshold=3) [...............................................XX..X........] duration: 684.801474ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/43834d9fac36e837 FAIL (2025-12-09 07:35:52 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 684.83029ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 1.980632118s trace: https://jaeger.dev.storj.io/trace/55634d9c1d55ff67 FAIL (2025-12-09 07:35:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.980663708s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.698120774s trace: https://jaeger.dev.storj.io/trace/219dd7469694ce8d FAIL (2025-12-09 07:35:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.221678783 (87h58m24.092761818s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222184665 (87h58m24.09326771s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222790810 (87h58m24.093873855s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223285452 (87h58m24.094368487s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223348001 (87h58m24.094431036s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223906905 (87h58m24.09498994s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224273312 (87h58m24.095356357s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224668005 (87h58m24.09575104s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225016438 (87h58m24.096099483s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225398876 (87h58m24.096481911s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225880544 (87h58m24.096963579s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.226416704 (87h58m24.097499739s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.227984600 (87h58m24.099067635s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.229456091 (87h58m24.100539136s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.231011924 (87h58m24.102094969s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.232660333 (87h58m24.103743388s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:25.694926842 (87h58m25.566009888s) [upload=76783]: Next returned nil segment Dec 9 07:27:25.694949827 (87h58m25.566032862s) [upload=76783]: Waiting for error group managing segments... Dec 9 07:27:26.134631394 (87h58m26.00571443s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.144638911 (87h58m26.015721956s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.194703193 (87h58m26.065786228s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.202448432 (87h58m26.073531477s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.218793957 (87h58m26.089876992s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.219708348 (87h58m26.090791393s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.227913963 (87h58m26.098996998s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.238868943 (87h58m26.109951988s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.249699476 (87h58m26.120782522s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.263815877 (87h58m26.134898912s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.264418876 (87h58m26.135501911s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.280320365 (87h58m26.1514034s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.294136013 (87h58m26.165219058s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321886986 (87h58m26.192970031s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321911793 (87h58m26.192994838s) [seg_pos={0 0}, upload=76783]: Segment reached optimal threshold of 14 pieces. Dec 9 07:27:26.322167190 (87h58m26.193250235s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322175145 (87h58m26.19325821s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.322770298 (87h58m26.193853333s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322779535 (87h58m26.19386258s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323079737 (87h58m26.194162792s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT): protocol: read tcp 188.166.225.170:43106->145.79.227.36:4548: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323089335 (87h58m26.19417238s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323372955 (87h58m26.19445599s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: read tcp 188.166.225.170:50502->64.49.13.169:4716: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323381462 (87h58m26.194464507s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323400197 (87h58m26.194483242s) [seg_pos={0 0}, upload=76783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:27:26.323490970 (87h58m26.194574015s) [upload=76783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:27:26.323497242 (87h58m26.194580287s) [upload=76783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:30:43.052543346 (88h1m42.923626391s) [upload=76783]: Done uploading segments... Dec 9 07:30:43.052668744 (88h1m42.923751799s) [upload=76783]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.698159609s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.............................................XXX............] duration: 2.191046566s trace: https://jaeger.dev.storj.io/trace/178b4cba623ae119 FAIL (2025-12-09 07:35:32 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.191071884s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 434.876179ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt trace: https://jaeger.dev.storj.io/trace/51a280a8635ce0b9 FAIL (2025-12-09 07:35:31 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 434.901237ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 4.873902527s trace: https://jaeger.dev.storj.io/trace/50f433cc1e60731c FAIL (2025-12-09 07:35:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.873924378s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 1.966166383s trace: https://jaeger.dev.storj.io/trace/42dcab7a5e240fd8 FAIL (2025-12-09 07:35:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:28:09.412200246 (87h59m9.283283281s) [upload=76792]: Starting upload Dec 9 07:28:09.412257335 (87h59m9.283340371s) [upload=76792]: Done starting upload Dec 9 07:28:09.412337338 (87h59m9.283420383s) [upload=76792]: Uploading segments... Dec 9 07:28:09.412384597 (87h59m9.283467632s) [upload=76792]: Got next segment. Inline: true Dec 9 07:28:09.412398334 (87h59m9.283481389s) [upload=76792]: Waiting for error group managing segments... Dec 9 07:28:09.412446225 (87h59m9.28352927s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:28:09.412452456 (87h59m9.283535491s) [upload=76792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:28:09.412455432 (87h59m9.283538477s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:28:19.867794892 (87h59m19.738877927s) [upload=76792]: Done uploading segments... Dec 9 07:28:19.867843895 (87h59m19.73892694s) [upload=76792]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.966187783s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 46.7537193s trace: https://jaeger.dev.storj.io/trace/50f433cc1e60731c FAIL (2025-12-09 07:35:24 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 46.753743566s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X...................X.................] duration: 5.791267117s trace: https://jaeger.dev.storj.io/trace/59c049b2e292a646 FAIL (2025-12-09 07:35:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.791298615s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 1.983233s trace: https://jaeger.dev.storj.io/trace/55634d9c1d55ff67 FAIL (2025-12-09 07:35:17 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.983260623s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [..............................................XXX...........] duration: 2.132314384s trace: https://jaeger.dev.storj.io/trace/178b4cba623ae119 FAIL (2025-12-09 07:35:13 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.132341055s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 3 out of 60 is failed (threshold=3) [................................................XX..X.......] duration: 683.297501ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/43834d9fac36e837 FAIL (2025-12-09 07:34:56 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 683.327649ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 1.943516776s trace: https://jaeger.dev.storj.io/trace/42dcab7a5e240fd8 FAIL (2025-12-09 07:34:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:28:09.412200246 (87h59m9.283283281s) [upload=76792]: Starting upload Dec 9 07:28:09.412257335 (87h59m9.283340371s) [upload=76792]: Done starting upload Dec 9 07:28:09.412337338 (87h59m9.283420383s) [upload=76792]: Uploading segments... Dec 9 07:28:09.412384597 (87h59m9.283467632s) [upload=76792]: Got next segment. Inline: true Dec 9 07:28:09.412398334 (87h59m9.283481389s) [upload=76792]: Waiting for error group managing segments... Dec 9 07:28:09.412446225 (87h59m9.28352927s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:28:09.412452456 (87h59m9.283535491s) [upload=76792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:28:09.412455432 (87h59m9.283538477s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:28:19.867794892 (87h59m19.738877927s) [upload=76792]: Done uploading segments... Dec 9 07:28:19.867843895 (87h59m19.73892694s) [upload=76792]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.94354618s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [.X...X...X............................X...........XX..X.....] duration: 5.591308913s trace: https://jaeger.dev.storj.io/trace/27a6ebfbc56e96c0 FAIL (2025-12-09 07:34:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.591343299s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................XX..........] duration: 6.153634099s trace: https://jaeger.dev.storj.io/trace/5f10a1154ce06783 FAIL (2025-12-09 07:34:47 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.153663385s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 439.108006ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt trace: https://jaeger.dev.storj.io/trace/51a280a8635ce0b9 FAIL (2025-12-09 07:34:42 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 439.135157ms edge linksharing us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...............................................X.XX.XX..X...] duration: 3.466825197s trace: https://jaeger.dev.storj.io/trace/402bfb4648c7d69b FAIL (2025-12-09 07:34:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.46685295s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X.................................XXX.X.......] duration: 5.55900861s trace: https://jaeger.dev.storj.io/trace/34e772a771ff36b2 FAIL (2025-12-09 07:34:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :29:22.029442363 (88h0m21.900525398s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.029471198 (88h0m21.900554233s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.030555563 (88h0m21.901638608s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031156617 (88h0m21.902239662s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031594381 (88h0m21.902677416s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032034670 (88h0m21.903117715s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032167963 (88h0m21.903250998s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032800087 (88h0m21.903883132s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:23.836496806 (88h0m23.707579841s) [upload=76810]: Next returned nil segment Dec 9 07:29:23.836512486 (88h0m23.707595531s) [upload=76810]: Waiting for error group managing segments... Dec 9 07:29:24.124325638 (88h0m23.995408683s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.114522896 (88h0m24.985605941s) [noise=true , address=145.79.227.21:4475 , piece_id=ED4VS3YEIFPJGGKW36BJKVLT3EXGVOFG3SD23LEPPFIAOOKHUD4Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.377083973 (88h0m25.248167008s) [noise=true , address=145.79.227.41:4086 , piece_id=EZUBNMEUWTNJBXJBWJYXFJQEPFFVQOUHZAWVJW6NCWKGRKAS2PSA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.480466423 (88h0m25.351549468s) [noise=true , address=145.79.227.42:4121 , piece_id=7I2TXNXZM2KM5TEDDF7Q5DDUSBAPBE3WWFTTJFGNM5JVVCTQNU5Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:26.767037853 (88h0m26.638120908s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:49.689337786 (88h0m49.560420841s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection Dec 9 07:29:49.689500304 (88h0m49.56058335s) [noise=true , address=145.79.227.34:4907 , piece_id=UG6M3T655YGW4OGBYEVOLQVR7U5XWXPDDNXBZAPTHYP2DYWDRRYA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection Dec 9 07:29:49.689541303 (88h0m49.560624348s) [noise=true , address=145.79.227.33:4870 , piece_id=6Q5XSDCJCDRWHPO7JWCDNPQM5JN2ESDCRXWY5V23AGA6Q4C5Y7HQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection Dec 9 07:29:49.689654439 (88h0m49.560737474s) [noise=true , address=145.79.227.29:4943 , piece_id=FQ2FNNQR6QBWTJOIQBIOU7BSWIA6ZZMDWC4DE42NUNR5Z7ZDGCGQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection Dec 9 07:29:49.689700015 (88h0m49.56078306s) [noise=true , address=145.79.227.42:4122 , piece_id=5WKTAWBKWCZVU3YX4RSF4FQHQABNCZUHTXMTK3AIHUAGYDQXQYVQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection Dec 9 07:29:49.689861212 (88h0m49.560944247s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection Dec 9 07:29:49.689914644 (88h0m49.560997689s) [noise=true , address=145.79.227.28:4725 , piece_id=AGDXVUHDFM7DC2HNW3YUKREAROODLZBIVUMTFYBY3CXYQ7HSYGWA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection Dec 9 07:29:49.690003323 (88h0m49.561086368s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection Dec 9 07:29:49.690090949 (88h0m49.561173994s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection Dec 9 07:29:49.690262546 (88h0m49.561345591s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection Dec 9 07:29:49.690582726 (88h0m49.561665772s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:50466->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:29:49.690631769 (88h0m49.561714805s) [seg_pos={0 0}, upload=76810]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Dec 9 07:29:50.737500571 (88h0m50.608583616s) [upload=76810]: Done uploading segments... Dec 9 07:29:50.737550175 (88h0m50.60863321s) [upload=76810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.559035041s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................XX.........] duration: 5.950990459s trace: https://jaeger.dev.storj.io/trace/5f10a1154ce06783 FAIL (2025-12-09 07:34:28 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.95101746s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.006920714s trace: https://jaeger.dev.storj.io/trace/42dcab7a5e240fd8 FAIL (2025-12-09 07:34:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 07:28:09.412200246 (87h59m9.283283281s) [upload=76792]: Starting upload Dec 9 07:28:09.412257335 (87h59m9.283340371s) [upload=76792]: Done starting upload Dec 9 07:28:09.412337338 (87h59m9.283420383s) [upload=76792]: Uploading segments... Dec 9 07:28:09.412384597 (87h59m9.283467632s) [upload=76792]: Got next segment. Inline: true Dec 9 07:28:09.412398334 (87h59m9.283481389s) [upload=76792]: Waiting for error group managing segments... Dec 9 07:28:09.412446225 (87h59m9.28352927s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 07:28:09.412452456 (87h59m9.283535491s) [upload=76792]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 07:28:09.412455432 (87h59m9.283538477s) [upload=76792]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:28:19.867794892 (87h59m19.738877927s) [upload=76792]: Done uploading segments... Dec 9 07:28:19.867843895 (87h59m19.73892694s) [upload=76792]: Upload finished. err: metaclient: context deadline exceeded Duration:* 2.006947175s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.607422648s trace: https://jaeger.dev.storj.io/trace/219dd7469694ce8d FAIL (2025-12-09 07:34:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.221678783 (87h58m24.092761818s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222184665 (87h58m24.09326771s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222790810 (87h58m24.093873855s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223285452 (87h58m24.094368487s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223348001 (87h58m24.094431036s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223906905 (87h58m24.09498994s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224273312 (87h58m24.095356357s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224668005 (87h58m24.09575104s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225016438 (87h58m24.096099483s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225398876 (87h58m24.096481911s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225880544 (87h58m24.096963579s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.226416704 (87h58m24.097499739s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.227984600 (87h58m24.099067635s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.229456091 (87h58m24.100539136s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.231011924 (87h58m24.102094969s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.232660333 (87h58m24.103743388s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:25.694926842 (87h58m25.566009888s) [upload=76783]: Next returned nil segment Dec 9 07:27:25.694949827 (87h58m25.566032862s) [upload=76783]: Waiting for error group managing segments... Dec 9 07:27:26.134631394 (87h58m26.00571443s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.144638911 (87h58m26.015721956s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.194703193 (87h58m26.065786228s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.202448432 (87h58m26.073531477s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.218793957 (87h58m26.089876992s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.219708348 (87h58m26.090791393s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.227913963 (87h58m26.098996998s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.238868943 (87h58m26.109951988s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.249699476 (87h58m26.120782522s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.263815877 (87h58m26.134898912s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.264418876 (87h58m26.135501911s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.280320365 (87h58m26.1514034s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.294136013 (87h58m26.165219058s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321886986 (87h58m26.192970031s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321911793 (87h58m26.192994838s) [seg_pos={0 0}, upload=76783]: Segment reached optimal threshold of 14 pieces. Dec 9 07:27:26.322167190 (87h58m26.193250235s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322175145 (87h58m26.19325821s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.322770298 (87h58m26.193853333s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322779535 (87h58m26.19386258s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323079737 (87h58m26.194162792s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT): protocol: read tcp 188.166.225.170:43106->145.79.227.36:4548: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323089335 (87h58m26.19417238s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323372955 (87h58m26.19445599s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: read tcp 188.166.225.170:50502->64.49.13.169:4716: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323381462 (87h58m26.194464507s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323400197 (87h58m26.194483242s) [seg_pos={0 0}, upload=76783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:27:26.323490970 (87h58m26.194574015s) [upload=76783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:27:26.323497242 (87h58m26.194580287s) [upload=76783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:30:43.052543346 (88h1m42.923626391s) [upload=76783]: Done uploading segments... Dec 9 07:30:43.052668744 (88h1m42.923751799s) [upload=76783]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.607448226s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...................X................] duration: 5.638820005s trace: https://jaeger.dev.storj.io/trace/59c049b2e292a646 FAIL (2025-12-09 07:34:16 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.63884958s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X.................................XXX.X......] duration: 5.680192024s trace: https://jaeger.dev.storj.io/trace/34e772a771ff36b2 FAIL (2025-12-09 07:34:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== :29:22.029442363 (88h0m21.900525398s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.029471198 (88h0m21.900554233s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.030555563 (88h0m21.901638608s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031156617 (88h0m21.902239662s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.031594381 (88h0m21.902677416s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032034670 (88h0m21.903117715s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032167963 (88h0m21.903250998s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:22.032800087 (88h0m21.903883132s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Uploading piece... Dec 9 07:29:23.836496806 (88h0m23.707579841s) [upload=76810]: Next returned nil segment Dec 9 07:29:23.836512486 (88h0m23.707595531s) [upload=76810]: Waiting for error group managing segments... Dec 9 07:29:24.124325638 (88h0m23.995408683s) [noise=true , address=77.90.31.10:4519 , piece_id=PLAP7Q4BYYOUCI53OQKNR74TS4XCPEI6GTH5CETYOWWCEPIDCTEA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.114522896 (88h0m24.985605941s) [noise=true , address=145.79.227.21:4475 , piece_id=ED4VS3YEIFPJGGKW36BJKVLT3EXGVOFG3SD23LEPPFIAOOKHUD4Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.377083973 (88h0m25.248167008s) [noise=true , address=145.79.227.41:4086 , piece_id=EZUBNMEUWTNJBXJBWJYXFJQEPFFVQOUHZAWVJW6NCWKGRKAS2PSA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:25.480466423 (88h0m25.351549468s) [noise=true , address=145.79.227.42:4121 , piece_id=7I2TXNXZM2KM5TEDDF7Q5DDUSBAPBE3WWFTTJFGNM5JVVCTQNU5Q, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:26.767037853 (88h0m26.638120908s) [noise=true , address=77.90.31.10:4520 , piece_id=5JGFSZCUORYTBSH5IEYGLGW3GJ7IIKZZBUNG7ZYI2DWKCAC7FN4A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: <nil> Dec 9 07:29:49.689337786 (88h0m49.560420841s) [noise=true , address=145.79.227.40:4049 , piece_id=OCQVF3EKPS273EYHIPTFDMIBPQF63OITQVS6RDUNLCOCRNWBS3EQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:145.79.227.40:4049): protocol: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35260->145.79.227.40:4049: use of closed network connection Dec 9 07:29:49.689500304 (88h0m49.56058335s) [noise=true , address=145.79.227.34:4907 , piece_id=UG6M3T655YGW4OGBYEVOLQVR7U5XWXPDDNXBZAPTHYP2DYWDRRYA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS, address:145.79.227.34:4907): protocol: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48142->145.79.227.34:4907: use of closed network connection Dec 9 07:29:49.689541303 (88h0m49.560624348s) [noise=true , address=145.79.227.33:4870 , piece_id=6Q5XSDCJCDRWHPO7JWCDNPQM5JN2ESDCRXWY5V23AGA6Q4C5Y7HQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:145.79.227.33:4870): protocol: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48964->145.79.227.33:4870: use of closed network connection Dec 9 07:29:49.689654439 (88h0m49.560737474s) [noise=true , address=145.79.227.29:4943 , piece_id=FQ2FNNQR6QBWTJOIQBIOU7BSWIA6ZZMDWC4DE42NUNR5Z7ZDGCGQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz, address:145.79.227.29:4943): protocol: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48622->145.79.227.29:4943: use of closed network connection Dec 9 07:29:49.689700015 (88h0m49.56078306s) [noise=true , address=145.79.227.42:4122 , piece_id=5WKTAWBKWCZVU3YX4RSF4FQHQABNCZUHTXMTK3AIHUAGYDQXQYVQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:145.79.227.42:4122): protocol: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53248->145.79.227.42:4122: use of closed network connection Dec 9 07:29:49.689861212 (88h0m49.560944247s) [noise=true , address=145.79.227.39:4014 , piece_id=QSO5DWCCBHO7WP2XJO2Y7RP3WMSJPIEZZWNKYNB6CHJNXXKTXUQQ, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:122mrzG2jkLjNdnrDVr6UMM2aw3wNNTaJMnNaUKdN8UwFUqLwVd, address:145.79.227.39:4014): protocol: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60706->145.79.227.39:4014: use of closed network connection Dec 9 07:29:49.689914644 (88h0m49.560997689s) [noise=true , address=145.79.227.28:4725 , piece_id=AGDXVUHDFM7DC2HNW3YUKREAROODLZBIVUMTFYBY3CXYQ7HSYGWA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1, address:145.79.227.28:4725): protocol: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34086->145.79.227.28:4725: use of closed network connection Dec 9 07:29:49.690003323 (88h0m49.561086368s) [noise=true , address=145.79.227.32:4834 , piece_id=LZGJJKA7LBTTBN26ZKVJKIMIQJDOXZVEIEUIL4Y73RPOFGJZOAOA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1gRnamgAa2ccZS6F44apRmzSkXC6jGvNreJ5J8Y9nciWPZCSsk, address:145.79.227.32:4834): protocol: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43002->145.79.227.32:4834: use of closed network connection Dec 9 07:29:49.690090949 (88h0m49.561173994s) [noise=true , address=145.79.227.26:4654 , piece_id=LH5BDOW3HG7P7VSLJ3EUBPLOPK3DPAXSJB5RMA5KFEMDQ3R52W3A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah, address:145.79.227.26:4654): protocol: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34578->145.79.227.26:4654: use of closed network connection Dec 9 07:29:49.690262546 (88h0m49.561345591s) [noise=true , address=145.79.227.2:4114 , piece_id=KNMNK52ESYIBOSEF4HTDJOFJSOXHXQWRUI5URRQ6WRQOFGWM47SA, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55674->145.79.227.2:4114: use of closed network connection Dec 9 07:29:49.690582726 (88h0m49.561665772s) [noise=true , address=145.79.227.20:4438 , piece_id=NC5ZFPAH575LULPHREYIQXSW44HHPJEUAVCM2RZYOACUWYOD6D6A, seg_pos={0 0}, upload=76810]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: read tcp 188.166.225.170:50466->145.79.227.20:4438: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:29:49.690631769 (88h0m49.561714805s) [seg_pos={0 0}, upload=76810]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Dec 9 07:29:50.737500571 (88h0m50.608583616s) [upload=76810]: Done uploading segments... Dec 9 07:29:50.737550175 (88h0m50.60863321s) [upload=76810]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 5.68021607s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.005397105s trace: https://jaeger.dev.storj.io/trace/55634d9c1d55ff67 FAIL (2025-12-09 07:34:13 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.005420739s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 3 out of 60 is failed (threshold=3) [...............................................XXX..........] duration: 2.250262494s trace: https://jaeger.dev.storj.io/trace/178b4cba623ae119 FAIL (2025-12-09 07:34:11 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded Duration: 2.250290708s edge gateway us-select-1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 3 out of 60 is failed (threshold=3) [.................................................XX..X......] duration: 681.377615ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/43834d9fac36e837 FAIL (2025-12-09 07:34:09 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 681.408454ms core satellite us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................XX........] duration: 5.840590998s trace: https://jaeger.dev.storj.io/trace/5f10a1154ce06783 FAIL (2025-12-09 07:34:08 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.840621005s edge gateway us-select-1
checkers.HTTP{"GET" "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt", ExpectedStatus: 308, ExpectedRedirect: "https://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt"} failure (antiblip window 3/60): (http.us_select_1.linksharing) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 444.24599ms target: http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt trace: https://jaeger.dev.storj.io/trace/51a280a8635ce0b9 FAIL (2025-12-09 07:34:06 UTC) orbiter warning: Get "http://link.us-select-1.storjshare.io/raw/jv6fvdi5muos6nw22tdksxgfrd2a/uptime-checks-us-select/hello-world.txt": dial tcp 45.38.24.2:80: i/o timeout Duration: 444.298801ms edge linksharing us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.71607297s trace: https://jaeger.dev.storj.io/trace/219dd7469694ce8d FAIL (2025-12-09 07:34:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.221678783 (87h58m24.092761818s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222184665 (87h58m24.09326771s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.222790810 (87h58m24.093873855s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223285452 (87h58m24.094368487s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223348001 (87h58m24.094431036s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.223906905 (87h58m24.09498994s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224273312 (87h58m24.095356357s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.224668005 (87h58m24.09575104s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225016438 (87h58m24.096099483s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225398876 (87h58m24.096481911s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.225880544 (87h58m24.096963579s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.226416704 (87h58m24.097499739s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.227984600 (87h58m24.099067635s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.229456091 (87h58m24.100539136s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.231011924 (87h58m24.102094969s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:24.232660333 (87h58m24.103743388s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Uploading piece... Dec 9 07:27:25.694926842 (87h58m25.566009888s) [upload=76783]: Next returned nil segment Dec 9 07:27:25.694949827 (87h58m25.566032862s) [upload=76783]: Waiting for error group managing segments... Dec 9 07:27:26.134631394 (87h58m26.00571443s) [noise=true , address=77.90.31.18:4810 , piece_id=KHDURLN4TI5CVATUNIPKXVZVE6RTSMYEXYRPQTTTI7HN5HPK4YQQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.144638911 (87h58m26.015721956s) [noise=true , address=145.79.227.52:4476 , piece_id=L6QZOLTNX3XYDXB666CVPSJGJXM6L6X77RSLRWEY4NHD7AZ5IAPQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.194703193 (87h58m26.065786228s) [noise=true , address=77.90.31.21:4893 , piece_id=OH7DDPDVFJD7IBUO5YM36ECLTRC4BTGDG67WBLKMC4YSCVZEBOPA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.202448432 (87h58m26.073531477s) [noise=true , address=45.91.116.7:4017 , piece_id=BCJDHWENGMC543NGCZXI45J7ZG55WBGEKT3UKBT63PA3J6ZUPU3A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.218793957 (87h58m26.089876992s) [noise=true , address=77.90.31.17:4785 , piece_id=4EJP5AEBHKSGGUW5ZPYEOISFAZELBXLOJFSAS3VV6OP7VLQ2F74A, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.219708348 (87h58m26.090791393s) [noise=true , address=77.90.31.16:4738 , piece_id=VRV6DRYVH4A2XSGAY3QXPCCUGFWB4QUIQGCDVHLFKW3LZDSTAPWQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.227913963 (87h58m26.098996998s) [noise=true , address=45.91.116.13:4177 , piece_id=2LU4SQLBTEWCDYPKFDKCPGDWDDJBGDVZCSDX2CB55G3K6H4QEY3Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.238868943 (87h58m26.109951988s) [noise=true , address=77.90.31.13:4633 , piece_id=KFBVBNGERLO7CAWRN6XRMC4VNGH7I4KTRBQVWJDVHGHPQCTYYRKQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.249699476 (87h58m26.120782522s) [noise=true , address=77.90.31.19:4846 , piece_id=YWZOHCBUBBAOCLACLHNN2PZMKFOUVMBOLDQQOWJBOTUNX2QLWUKA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.263815877 (87h58m26.134898912s) [noise=true , address=77.90.31.14:4665 , piece_id=CCWLWRFY5WZXQUNHY2XZBWACBBEFJP3ZLENPBFCQ2B3OUADQ4HBA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.264418876 (87h58m26.135501911s) [noise=true , address=45.91.116.9:4269 , piece_id=DCNNMUUWBVQQGHO2X2ONNRYXC6APCOTZ7FZY3SOEKGVM2PXUP5LA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.280320365 (87h58m26.1514034s) [noise=true , address=77.90.31.12:4596 , piece_id=JMJOAO3IGDT3TTZQYT435VS5ULMF6W4LM7IG4OCZTXYWFAMJRN6Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.294136013 (87h58m26.165219058s) [noise=true , address=45.91.116.11:4143 , piece_id=DHHWMYDREKIXD7ZBE65APW57VWVIGPTMGNZBOOJXIGJXQHTEK7GA, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321886986 (87h58m26.192970031s) [noise=true , address=77.90.31.15:4703 , piece_id=GUTSQVQOTKF32EUSRUFTCPPCEE276DBDJH43CC3UN7LUIRWFPHFQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: <nil> Dec 9 07:27:26.321911793 (87h58m26.192994838s) [seg_pos={0 0}, upload=76783]: Segment reached optimal threshold of 14 pieces. Dec 9 07:27:26.322167190 (87h58m26.193250235s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ke332kYFGCHfGQ7hf5uDwZK7BZJhuDXktzpC842JXqE14S9mL): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322175145 (87h58m26.19325821s) [noise=true , address=45.91.116.6:4996 , piece_id=LFXHWMRRXGNQMQT5DGXK3JBB5WDCX2VCPQ4GD5PYJ3ZA25RU3G7Q, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.322770298 (87h58m26.193853333s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5): protocol: expected piece hash; context canceled; EOF Dec 9 07:27:26.322779535 (87h58m26.19386258s) [noise=true , address=145.79.227.30:4770 , piece_id=YIH2QPGUHNNBRTBKMRJUXUNWV5GV45V4Z4DDPYN77S656ZMOSYNQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323079737 (87h58m26.194162792s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P7nZqYyfKc2CT4Sf8bWjEM6zk34vVz9UMifuKRybUjWKQqekT): protocol: read tcp 188.166.225.170:43106->145.79.227.36:4548: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323089335 (87h58m26.19417238s) [noise=true , address=145.79.227.36:4548 , piece_id=PSSJELITXEJ4BLVXN2ZLYMZLEOSVVFUOTPMIN3UI6WGGBRILAORQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323372955 (87h58m26.19445599s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: read tcp 188.166.225.170:50502->64.49.13.169:4716: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 9 07:27:26.323381462 (87h58m26.194464507s) [noise=true , address=64.49.13.169:4716 , piece_id=42IYD2OI2S5U5BILXUZB5GW75PURFHINAKSX4YDUGHSDE2TBR6ZQ, seg_pos={0 0}, upload=76783]: Long tail cancellation due to optimal threshold Dec 9 07:27:26.323400197 (87h58m26.194483242s) [seg_pos={0 0}, upload=76783]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 9 07:27:26.323490970 (87h58m26.194574015s) [upload=76783]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 07:27:26.323497242 (87h58m26.194580287s) [upload=76783]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:30:43.052543346 (88h1m42.923626391s) [upload=76783]: Done uploading segments... Dec 9 07:30:43.052668744 (88h1m42.923751799s) [upload=76783]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.7160997s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)