Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X............X..........................] duration: 47.816237424s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:23:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 47.816276559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [...XXXX...XXXX...X.X...XX..X.X..XX.......XX..X.X...X.X..X.X.] duration: 5.984902097s trace: https://jaeger.dev.storj.io/trace/6aa68c927e11 FAIL (2025-12-30 04:23:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: SKS5JZDRLFYZQK5SKRBFCYVDTPS6ND4OPEANVHZQBRF64TVPWKVQ) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: E2ISEI3O7YW7SXGFE4NNB6YWERXTZUT6DQZOX5K2DL2UYQHW2U7A) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: TILISC65JUZKTVZ532LEBFY6SIBV45FI3H3QMVXAKYOOBEYRLTBQ) context deadline exceeded ======LOGS====== 30 04:22:48.866177679 (588h53m48.737260714s) [upload=516248]: Starting upload Dec 30 04:22:48.866229428 (588h53m48.737312463s) [upload=516248]: Done starting upload Dec 30 04:22:48.866375335 (588h53m48.73745838s) [upload=516248]: Uploading segments... Dec 30 04:22:48.866576719 (588h53m48.737659754s) [upload=516248]: Got next segment. Inline: false Dec 30 04:22:48.866611565 (588h53m48.737694611s) [upload=516248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:22:48.866616996 (588h53m48.737700031s) [upload=516248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:22:49.115191553 (588h53m48.986274598s) [seg_pos={0 0}, upload=516248]: Begin upload segment... Dec 30 04:22:49.115238923 (588h53m48.986321958s) [seg_pos={0 0}, upload=516248]: Done begin upload segment. Dec 30 04:22:49.115268489 (588h53m48.986351524s) [noise=true , address=45.91.116.14:4189 , piece_id=SKS5JZDRLFYZQK5SKRBFCYVDTPS6ND4OPEANVHZQBRF64TVPWKVQ, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.118398098 (588h53m48.989481174s) [noise=true , address=77.90.31.11:4556 , piece_id=UVEIP45PRZV4KDMW6X3BZ4YLQ2YHM4TOGOQQ4EGOMSM6PZ7FM7QQ, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.119409655 (588h53m48.99049269s) [noise=true , address=77.90.31.10:4517 , piece_id=ITSR5AF2RZPOZWD7AOGUOYUNHJOFSMXWR7MATPYVOVRDYYVM7FXQ, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.120490934 (588h53m48.991573969s) [noise=true , address=77.90.31.18:4808 , piece_id=E2ISEI3O7YW7SXGFE4NNB6YWERXTZUT6DQZOX5K2DL2UYQHW2U7A, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.120954357 (588h53m48.992037402s) [noise=true , address=145.79.227.28:4726 , piece_id=MJMOF2N2AYPRVMSWQEHMGIWGS4YPHLPBN2IVTC6GZS6LUHXZKJAA, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.122212232 (588h53m48.993295267s) [noise=true , address=145.79.227.31:4799 , piece_id=563VV4EVX5KJ2SW42M5FG4UUJ2OGXG2KILHVT3DH63XFJZKOR4IQ, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.122540066 (588h53m48.993623132s) [noise=true , address=77.90.31.16:4735 , piece_id=TILISC65JUZKTVZ532LEBFY6SIBV45FI3H3QMVXAKYOOBEYRLTBQ, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.124120435 (588h53m48.99520347s) [noise=true , address=145.79.227.15:4875 , piece_id=YD5CVSLNY4NMVU33URG6T46JPUTDQSEAWVYKTE4MTHRQ3JYCUZGA, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.124572606 (588h53m48.995655641s) [noise=true , address=145.79.227.20:4438 , piece_id=4ASMGBNBR5MJ6RIDGWDFGLTDVB57VTK5LD3PIHTTDH77BP2BOC3A, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.125511946 (588h53m48.996594981s) [noise=true , address=145.79.227.7:4293 , piece_id=LYYQ2H6DAAQQTLVWKPXFUQULGHNOOLH5F7ERXTFBFHKV44XDYCAA, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.126936609 (588h53m48.998019654s) [noise=true , address=145.79.227.19:4079 , piece_id=F5ORFUBHODYREDYUGXTPDCQBV2GJJBXQ4HRPIS576GP5JPDFG7UA, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.127100540 (588h53m48.998183585s) [noise=true , address=45.91.116.13:4156 , piece_id=5BINX5GN2PYKAYDSKHCMH47OJP6UPHK73VGIYWQMV2E5UJUR2FGQ, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.128733831 (588h53m48.999816866s) [noise=true , address=145.79.227.3:4149 , piece_id=ED5OEHOFC6YQZ6ZW37EGVRWI3NAQRTYTUCNUHMMMQZPZDJMGSH3A, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.128846205 (588h53m48.99992924s) [noise=true , address=45.91.116.7:4009 , piece_id=EHXAWYVX7HHGQRBA4JAQMTNAURSLGMQCEP6P4D5KZFPO6EBNUDZQ, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.131045894 (588h53m49.002129019s) [noise=true , address=145.79.227.25:4619 , piece_id=C4QZZW2M7WLB73D3733LDX3GITII3GHRTTVR63JVOI7MPCBXVNBQ, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.135845022 (588h53m49.006928067s) [noise=true , address=45.91.116.9:4262 , piece_id=VXDIIUFC26KKNXLCOS72OQJL7T6QE52NVBBK6TFBXK6YFXJ7HZDQ, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:50.823839093 (588h53m50.694922128s) [upload=516248]: Next returned nil segment Dec 30 04:22:50.823859463 (588h53m50.694942498s) [upload=516248]: Waiting for error group managing segments... Dec 30 04:22:51.103693556 (588h53m50.974776601s) [noise=true , address=145.79.227.19:4079 , piece_id=F5ORFUBHODYREDYUGXTPDCQBV2GJJBXQ4HRPIS576GP5JPDFG7UA, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.111582239 (588h53m50.982665274s) [noise=true , address=145.79.227.3:4149 , piece_id=ED5OEHOFC6YQZ6ZW37EGVRWI3NAQRTYTUCNUHMMMQZPZDJMGSH3A, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.130035539 (588h53m51.001118574s) [noise=true , address=145.79.227.7:4293 , piece_id=LYYQ2H6DAAQQTLVWKPXFUQULGHNOOLH5F7ERXTFBFHKV44XDYCAA, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.143650024 (588h53m51.014733069s) [noise=true , address=145.79.227.15:4875 , piece_id=YD5CVSLNY4NMVU33URG6T46JPUTDQSEAWVYKTE4MTHRQ3JYCUZGA, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.144090493 (588h53m51.015173528s) [noise=true , address=145.79.227.31:4799 , piece_id=563VV4EVX5KJ2SW42M5FG4UUJ2OGXG2KILHVT3DH63XFJZKOR4IQ, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.144124298 (588h53m51.015207353s) [noise=true , address=145.79.227.20:4438 , piece_id=4ASMGBNBR5MJ6RIDGWDFGLTDVB57VTK5LD3PIHTTDH77BP2BOC3A, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.144444628 (588h53m51.015527673s) [noise=true , address=145.79.227.28:4726 , piece_id=MJMOF2N2AYPRVMSWQEHMGIWGS4YPHLPBN2IVTC6GZS6LUHXZKJAA, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.146140627 (588h53m51.017223662s) [noise=true , address=145.79.227.25:4619 , piece_id=C4QZZW2M7WLB73D3733LDX3GITII3GHRTTVR63JVOI7MPCBXVNBQ, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.373651784 (588h53m51.244734829s) [noise=true , address=45.91.116.14:4189 , piece_id=SKS5JZDRLFYZQK5SKRBFCYVDTPS6ND4OPEANVHZQBRF64TVPWKVQ, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.376507322 (588h53m51.247590367s) [noise=true , address=77.90.31.16:4735 , piece_id=TILISC65JUZKTVZ532LEBFY6SIBV45FI3H3QMVXAKYOOBEYRLTBQ, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.393862151 (588h53m51.264945196s) [noise=true , address=77.90.31.18:4808 , piece_id=E2ISEI3O7YW7SXGFE4NNB6YWERXTZUT6DQZOX5K2DL2UYQHW2U7A, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.423720248 (588h53m51.294803293s) [noise=true , address=77.90.31.11:4556 , piece_id=UVEIP45PRZV4KDMW6X3BZ4YLQ2YHM4TOGOQQ4EGOMSM6PZ7FM7QQ, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.485078592 (588h53m51.356161627s) [noise=true , address=45.91.116.13:4156 , piece_id=5BINX5GN2PYKAYDSKHCMH47OJP6UPHK73VGIYWQMV2E5UJUR2FGQ, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.485100194 (588h53m51.356183229s) [seg_pos={0 0}, upload=516248]: Segment reached optimal threshold of 13 pieces. Dec 30 04:22:51.485309352 (588h53m51.356392387s) [noise=true , address=45.91.116.9:4262 , piece_id=VXDIIUFC26KKNXLCOS72OQJL7T6QE52NVBBK6TFBXK6YFXJ7HZDQ, seg_pos={0 0}, upload=516248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 30 04:22:51.485317307 (588h53m51.356400352s) [noise=true , address=45.91.116.9:4262 , piece_id=VXDIIUFC26KKNXLCOS72OQJL7T6QE52NVBBK6TFBXK6YFXJ7HZDQ, seg_pos={0 0}, upload=516248]: Long tail cancellation due to optimal threshold Dec 30 04:22:51.485444309 (588h53m51.356527354s) [noise=true , address=45.91.116.7:4009 , piece_id=EHXAWYVX7HHGQRBA4JAQMTNAURSLGMQCEP6P4D5KZFPO6EBNUDZQ, seg_pos={0 0}, upload=516248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 04:22:51.485460189 (588h53m51.356543234s) [noise=true , address=45.91.116.7:4009 , piece_id=EHXAWYVX7HHGQRBA4JAQMTNAURSLGMQCEP6P4D5KZFPO6EBNUDZQ, seg_pos={0 0}, upload=516248]: Long tail cancellation due to optimal threshold Dec 30 04:22:51.485608521 (588h53m51.356691556s) [noise=true , address=77.90.31.10:4517 , piece_id=ITSR5AF2RZPOZWD7AOGUOYUNHJOFSMXWR7MATPYVOVRDYYVM7FXQ, seg_pos={0 0}, upload=516248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:42368->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:42368->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42368->77.90.31.10:4517: use of closed network connection Dec 30 04:22:51.485618250 (588h53m51.356701285s) [noise=true , address=77.90.31.10:4517 , piece_id=ITSR5AF2RZPOZWD7AOGUOYUNHJOFSMXWR7MATPYVOVRDYYVM7FXQ, seg_pos={0 0}, upload=516248]: Long tail cancellation due to optimal threshold Dec 30 04:22:51.485635132 (588h53m51.356718167s) [seg_pos={0 0}, upload=516248]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:22:51.485703793 (588h53m51.356786828s) [upload=516248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:22:51.485708942 (588h53m51.356791977s) [upload=516248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:22:51.772206877 (588h53m51.643289942s) [upload=516248]: Done uploading segments... Dec 30 04:22:51.772224560 (588h53m51.643307605s) [upload=516248]: Upload finished. err: <nil> Duration: 5.984923518s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X...XXXX...XXXX...X.X...XX..X.X..XX.......XX..X.X...X.X..X.X] duration: 30.003361276s trace: https://jaeger.dev.storj.io/trace/6aa68c927e11 FAIL (2025-12-30 04:23:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: SKS5JZDRLFYZQK5SKRBFCYVDTPS6ND4OPEANVHZQBRF64TVPWKVQ) context deadline exceeded; piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: E2ISEI3O7YW7SXGFE4NNB6YWERXTZUT6DQZOX5K2DL2UYQHW2U7A) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: TILISC65JUZKTVZ532LEBFY6SIBV45FI3H3QMVXAKYOOBEYRLTBQ) context deadline exceeded ======LOGS====== 30 04:22:48.866177679 (588h53m48.737260714s) [upload=516248]: Starting upload Dec 30 04:22:48.866229428 (588h53m48.737312463s) [upload=516248]: Done starting upload Dec 30 04:22:48.866375335 (588h53m48.73745838s) [upload=516248]: Uploading segments... Dec 30 04:22:48.866576719 (588h53m48.737659754s) [upload=516248]: Got next segment. Inline: false Dec 30 04:22:48.866611565 (588h53m48.737694611s) [upload=516248]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:22:48.866616996 (588h53m48.737700031s) [upload=516248]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:22:49.115191553 (588h53m48.986274598s) [seg_pos={0 0}, upload=516248]: Begin upload segment... Dec 30 04:22:49.115238923 (588h53m48.986321958s) [seg_pos={0 0}, upload=516248]: Done begin upload segment. Dec 30 04:22:49.115268489 (588h53m48.986351524s) [noise=true , address=45.91.116.14:4189 , piece_id=SKS5JZDRLFYZQK5SKRBFCYVDTPS6ND4OPEANVHZQBRF64TVPWKVQ, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.118398098 (588h53m48.989481174s) [noise=true , address=77.90.31.11:4556 , piece_id=UVEIP45PRZV4KDMW6X3BZ4YLQ2YHM4TOGOQQ4EGOMSM6PZ7FM7QQ, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.119409655 (588h53m48.99049269s) [noise=true , address=77.90.31.10:4517 , piece_id=ITSR5AF2RZPOZWD7AOGUOYUNHJOFSMXWR7MATPYVOVRDYYVM7FXQ, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.120490934 (588h53m48.991573969s) [noise=true , address=77.90.31.18:4808 , piece_id=E2ISEI3O7YW7SXGFE4NNB6YWERXTZUT6DQZOX5K2DL2UYQHW2U7A, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.120954357 (588h53m48.992037402s) [noise=true , address=145.79.227.28:4726 , piece_id=MJMOF2N2AYPRVMSWQEHMGIWGS4YPHLPBN2IVTC6GZS6LUHXZKJAA, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.122212232 (588h53m48.993295267s) [noise=true , address=145.79.227.31:4799 , piece_id=563VV4EVX5KJ2SW42M5FG4UUJ2OGXG2KILHVT3DH63XFJZKOR4IQ, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.122540066 (588h53m48.993623132s) [noise=true , address=77.90.31.16:4735 , piece_id=TILISC65JUZKTVZ532LEBFY6SIBV45FI3H3QMVXAKYOOBEYRLTBQ, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.124120435 (588h53m48.99520347s) [noise=true , address=145.79.227.15:4875 , piece_id=YD5CVSLNY4NMVU33URG6T46JPUTDQSEAWVYKTE4MTHRQ3JYCUZGA, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.124572606 (588h53m48.995655641s) [noise=true , address=145.79.227.20:4438 , piece_id=4ASMGBNBR5MJ6RIDGWDFGLTDVB57VTK5LD3PIHTTDH77BP2BOC3A, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.125511946 (588h53m48.996594981s) [noise=true , address=145.79.227.7:4293 , piece_id=LYYQ2H6DAAQQTLVWKPXFUQULGHNOOLH5F7ERXTFBFHKV44XDYCAA, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.126936609 (588h53m48.998019654s) [noise=true , address=145.79.227.19:4079 , piece_id=F5ORFUBHODYREDYUGXTPDCQBV2GJJBXQ4HRPIS576GP5JPDFG7UA, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.127100540 (588h53m48.998183585s) [noise=true , address=45.91.116.13:4156 , piece_id=5BINX5GN2PYKAYDSKHCMH47OJP6UPHK73VGIYWQMV2E5UJUR2FGQ, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.128733831 (588h53m48.999816866s) [noise=true , address=145.79.227.3:4149 , piece_id=ED5OEHOFC6YQZ6ZW37EGVRWI3NAQRTYTUCNUHMMMQZPZDJMGSH3A, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.128846205 (588h53m48.99992924s) [noise=true , address=45.91.116.7:4009 , piece_id=EHXAWYVX7HHGQRBA4JAQMTNAURSLGMQCEP6P4D5KZFPO6EBNUDZQ, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.131045894 (588h53m49.002129019s) [noise=true , address=145.79.227.25:4619 , piece_id=C4QZZW2M7WLB73D3733LDX3GITII3GHRTTVR63JVOI7MPCBXVNBQ, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:49.135845022 (588h53m49.006928067s) [noise=true , address=45.91.116.9:4262 , piece_id=VXDIIUFC26KKNXLCOS72OQJL7T6QE52NVBBK6TFBXK6YFXJ7HZDQ, seg_pos={0 0}, upload=516248]: Uploading piece... Dec 30 04:22:50.823839093 (588h53m50.694922128s) [upload=516248]: Next returned nil segment Dec 30 04:22:50.823859463 (588h53m50.694942498s) [upload=516248]: Waiting for error group managing segments... Dec 30 04:22:51.103693556 (588h53m50.974776601s) [noise=true , address=145.79.227.19:4079 , piece_id=F5ORFUBHODYREDYUGXTPDCQBV2GJJBXQ4HRPIS576GP5JPDFG7UA, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.111582239 (588h53m50.982665274s) [noise=true , address=145.79.227.3:4149 , piece_id=ED5OEHOFC6YQZ6ZW37EGVRWI3NAQRTYTUCNUHMMMQZPZDJMGSH3A, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.130035539 (588h53m51.001118574s) [noise=true , address=145.79.227.7:4293 , piece_id=LYYQ2H6DAAQQTLVWKPXFUQULGHNOOLH5F7ERXTFBFHKV44XDYCAA, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.143650024 (588h53m51.014733069s) [noise=true , address=145.79.227.15:4875 , piece_id=YD5CVSLNY4NMVU33URG6T46JPUTDQSEAWVYKTE4MTHRQ3JYCUZGA, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.144090493 (588h53m51.015173528s) [noise=true , address=145.79.227.31:4799 , piece_id=563VV4EVX5KJ2SW42M5FG4UUJ2OGXG2KILHVT3DH63XFJZKOR4IQ, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.144124298 (588h53m51.015207353s) [noise=true , address=145.79.227.20:4438 , piece_id=4ASMGBNBR5MJ6RIDGWDFGLTDVB57VTK5LD3PIHTTDH77BP2BOC3A, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.144444628 (588h53m51.015527673s) [noise=true , address=145.79.227.28:4726 , piece_id=MJMOF2N2AYPRVMSWQEHMGIWGS4YPHLPBN2IVTC6GZS6LUHXZKJAA, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.146140627 (588h53m51.017223662s) [noise=true , address=145.79.227.25:4619 , piece_id=C4QZZW2M7WLB73D3733LDX3GITII3GHRTTVR63JVOI7MPCBXVNBQ, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.373651784 (588h53m51.244734829s) [noise=true , address=45.91.116.14:4189 , piece_id=SKS5JZDRLFYZQK5SKRBFCYVDTPS6ND4OPEANVHZQBRF64TVPWKVQ, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.376507322 (588h53m51.247590367s) [noise=true , address=77.90.31.16:4735 , piece_id=TILISC65JUZKTVZ532LEBFY6SIBV45FI3H3QMVXAKYOOBEYRLTBQ, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.393862151 (588h53m51.264945196s) [noise=true , address=77.90.31.18:4808 , piece_id=E2ISEI3O7YW7SXGFE4NNB6YWERXTZUT6DQZOX5K2DL2UYQHW2U7A, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.423720248 (588h53m51.294803293s) [noise=true , address=77.90.31.11:4556 , piece_id=UVEIP45PRZV4KDMW6X3BZ4YLQ2YHM4TOGOQQ4EGOMSM6PZ7FM7QQ, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.485078592 (588h53m51.356161627s) [noise=true , address=45.91.116.13:4156 , piece_id=5BINX5GN2PYKAYDSKHCMH47OJP6UPHK73VGIYWQMV2E5UJUR2FGQ, seg_pos={0 0}, upload=516248]: Done uploading piece. err: <nil> Dec 30 04:22:51.485100194 (588h53m51.356183229s) [seg_pos={0 0}, upload=516248]: Segment reached optimal threshold of 13 pieces. Dec 30 04:22:51.485309352 (588h53m51.356392387s) [noise=true , address=45.91.116.9:4262 , piece_id=VXDIIUFC26KKNXLCOS72OQJL7T6QE52NVBBK6TFBXK6YFXJ7HZDQ, seg_pos={0 0}, upload=516248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF Dec 30 04:22:51.485317307 (588h53m51.356400352s) [noise=true , address=45.91.116.9:4262 , piece_id=VXDIIUFC26KKNXLCOS72OQJL7T6QE52NVBBK6TFBXK6YFXJ7HZDQ, seg_pos={0 0}, upload=516248]: Long tail cancellation due to optimal threshold Dec 30 04:22:51.485444309 (588h53m51.356527354s) [noise=true , address=45.91.116.7:4009 , piece_id=EHXAWYVX7HHGQRBA4JAQMTNAURSLGMQCEP6P4D5KZFPO6EBNUDZQ, seg_pos={0 0}, upload=516248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 30 04:22:51.485460189 (588h53m51.356543234s) [noise=true , address=45.91.116.7:4009 , piece_id=EHXAWYVX7HHGQRBA4JAQMTNAURSLGMQCEP6P4D5KZFPO6EBNUDZQ, seg_pos={0 0}, upload=516248]: Long tail cancellation due to optimal threshold Dec 30 04:22:51.485608521 (588h53m51.356691556s) [noise=true , address=77.90.31.10:4517 , piece_id=ITSR5AF2RZPOZWD7AOGUOYUNHJOFSMXWR7MATPYVOVRDYYVM7FXQ, seg_pos={0 0}, upload=516248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: write tcp 188.166.225.170:42368->77.90.31.10:4517: use of closed network connection; write tcp 188.166.225.170:42368->77.90.31.10:4517: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42368->77.90.31.10:4517: use of closed network connection Dec 30 04:22:51.485618250 (588h53m51.356701285s) [noise=true , address=77.90.31.10:4517 , piece_id=ITSR5AF2RZPOZWD7AOGUOYUNHJOFSMXWR7MATPYVOVRDYYVM7FXQ, seg_pos={0 0}, upload=516248]: Long tail cancellation due to optimal threshold Dec 30 04:22:51.485635132 (588h53m51.356718167s) [seg_pos={0 0}, upload=516248]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:22:51.485703793 (588h53m51.356786828s) [upload=516248]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:22:51.485708942 (588h53m51.356791977s) [upload=516248]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:22:51.772206877 (588h53m51.643289942s) [upload=516248]: Done uploading segments... Dec 30 04:22:51.772224560 (588h53m51.643307605s) [upload=516248]: Upload finished. err: <nil> Duration: 30.003389772s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...............X..XXX............XX..X.....X..............X.] duration: 3.720503788s trace: https://jaeger.dev.storj.io/trace/2d35b3e0e1d8945d FAIL (2025-12-30 04:22:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.720525719s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X............X.........................] duration: 45.522396213s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:22:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 45.52242062s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [................X..XXX............XX..X.....X..............X] duration: 5.000690437s trace: https://jaeger.dev.storj.io/trace/2d35b3e0e1d8945d FAIL (2025-12-30 04:22:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000711137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX...XXXX...XXXX...X.X...XX..X.X..XX.......XX..X.X...X.X..X.] duration: 6.023921623s trace: https://jaeger.dev.storj.io/trace/455d6de5426ec2ab FAIL (2025-12-30 04:22:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 04:21:30.007567269 (588h52m29.878650314s) [upload=516232]: Starting upload Dec 30 04:21:30.007641130 (588h52m29.878724165s) [upload=516232]: Done starting upload Dec 30 04:21:30.007779433 (588h52m29.878862468s) [upload=516232]: Uploading segments... Dec 30 04:21:30.007885655 (588h52m29.8789687s) [upload=516232]: Got next segment. Inline: false Dec 30 04:21:30.007929128 (588h52m29.879012183s) [upload=516232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:21:30.007934989 (588h52m29.879018024s) [upload=516232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:21:30.253393762 (588h52m30.124476798s) [seg_pos={0 0}, upload=516232]: Begin upload segment... Dec 30 04:21:30.253440952 (588h52m30.124523997s) [seg_pos={0 0}, upload=516232]: Done begin upload segment. Dec 30 04:21:30.253477732 (588h52m30.124560797s) [noise=true , address=145.79.227.7:4293 , piece_id=6D2Y5RNCDDQZ5NZGVVAMY64NDMXAC5G7ZQBN2GI6LJUMRLSKLPYA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.253902430 (588h52m30.124985476s) [noise=true , address=145.79.227.20:4439 , piece_id=P6N5ZPK3CXFBUL4J6H6ZRUQ3Y322XNDTX6UQZOVQYFYKKH433E2Q, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.254328202 (588h52m30.125411237s) [noise=true , address=45.91.116.9:4261 , piece_id=F5ZJ52QY2UZWOTQWKTBRSSACRRM4UBOMGJ3HIZKAWNTTBO3TFPQA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.254869542 (588h52m30.125952577s) [noise=true , address=45.91.116.10:4081 , piece_id=2EHXRG6VDQRWBYNMTTFZKZVRC5WQSTFQ2YS6KDGTAKRN3WYJV5WA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.255276017 (588h52m30.126359052s) [noise=true , address=145.79.227.5:4222 , piece_id=SARA7UO4QZ7DZEKROQLDYBUQVHOWEEWQUMFECMF3CXJG3K5CUXEA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.255392348 (588h52m30.126475383s) [noise=true , address=77.90.31.13:4626 , piece_id=7JQWFFWMUTOQC6UOPEY33UW6CT6VB7KYSX7OJOJLHAB6MMLDDFJQ, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.255652634 (588h52m30.126735669s) [noise=true , address=77.90.31.18:4805 , piece_id=JAHULSRJ4CQX2CVM7FRDYU42QP65MDYZMEDA67FAU5TC5XH63KSA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.255886740 (588h52m30.126969785s) [noise=true , address=45.91.116.13:4155 , piece_id=CE7Z2YRR33ISP2FVWKNZ7NG2NK3JTFANPWFNPRXYRKX7MN7I6Z3A, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.256068315 (588h52m30.127151351s) [noise=true , address=145.79.227.25:4619 , piece_id=VQIXL5MALTMYE6ZTR5ZT5XPS6MJTVYEASB4B7ARADAVASWBODW7Q, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.256243800 (588h52m30.127326835s) [noise=true , address=145.79.227.22:4509 , piece_id=NGXZOJLBEEJLJLAGBPS2UPWELZKMQMYQB2RKKGUTOSK5F4X6AKOA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.256475631 (588h52m30.127558666s) [noise=true , address=77.90.31.21:4878 , piece_id=NCT377CAKGUOSA4TE2VZ5BUUH5P763ARAVCGLL3AZZWDM5264FHA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.256643480 (588h52m30.127726515s) [noise=true , address=145.79.227.36:4517 , piece_id=WX3UKKPRD6TXM5C3RT66LT34VQILJTMXPUVMRLLU2MMNCJV4HPXA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.256923003 (588h52m30.128006048s) [noise=true , address=145.79.227.11:4730 , piece_id=DN4OPI2FP7K2A4XNQPT7U6A5GZ6WQXRLHN6R5JKB7WQITTSJNZAA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.257027132 (588h52m30.128110177s) [noise=true , address=45.91.116.14:4190 , piece_id=GNHZLIUFUWIU75NATSTLAZYPCOADGS5SQOG7PDWYOFIEAHKLA7RQ, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.257374593 (588h52m30.128457638s) [noise=true , address=45.91.116.11:4119 , piece_id=IKGHK5WI7SAAYUEMAMZVNNSIIQUCY2DY4XEP5WZZ56JOXC7AE6DA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.261062376 (588h52m30.132145421s) [noise=true , address=145.79.227.9:4367 , piece_id=MBVV7CQIKR6RYELQ7DYI42BJ75A5DHYXOHZSQENA5YTDHEI5TXQQ, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:31.870217596 (588h52m31.741300641s) [upload=516232]: Next returned nil segment Dec 30 04:21:31.870375816 (588h52m31.741458872s) [upload=516232]: Waiting for error group managing segments... Dec 30 04:21:32.250613077 (588h52m32.121696122s) [noise=true , address=145.79.227.22:4509 , piece_id=NGXZOJLBEEJLJLAGBPS2UPWELZKMQMYQB2RKKGUTOSK5F4X6AKOA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.252727573 (588h52m32.123810618s) [noise=true , address=145.79.227.36:4517 , piece_id=WX3UKKPRD6TXM5C3RT66LT34VQILJTMXPUVMRLLU2MMNCJV4HPXA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.261585852 (588h52m32.132668887s) [noise=true , address=145.79.227.20:4439 , piece_id=P6N5ZPK3CXFBUL4J6H6ZRUQ3Y322XNDTX6UQZOVQYFYKKH433E2Q, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.263402672 (588h52m32.134485717s) [noise=true , address=145.79.227.7:4293 , piece_id=6D2Y5RNCDDQZ5NZGVVAMY64NDMXAC5G7ZQBN2GI6LJUMRLSKLPYA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.266322282 (588h52m32.137405317s) [noise=true , address=145.79.227.25:4619 , piece_id=VQIXL5MALTMYE6ZTR5ZT5XPS6MJTVYEASB4B7ARADAVASWBODW7Q, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.267473644 (588h52m32.138556689s) [noise=true , address=145.79.227.5:4222 , piece_id=SARA7UO4QZ7DZEKROQLDYBUQVHOWEEWQUMFECMF3CXJG3K5CUXEA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.271410441 (588h52m32.142493496s) [noise=true , address=145.79.227.11:4730 , piece_id=DN4OPI2FP7K2A4XNQPT7U6A5GZ6WQXRLHN6R5JKB7WQITTSJNZAA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.286605166 (588h52m32.157688201s) [noise=true , address=145.79.227.9:4367 , piece_id=MBVV7CQIKR6RYELQ7DYI42BJ75A5DHYXOHZSQENA5YTDHEI5TXQQ, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.508009643 (588h52m32.379092678s) [noise=true , address=45.91.116.13:4155 , piece_id=CE7Z2YRR33ISP2FVWKNZ7NG2NK3JTFANPWFNPRXYRKX7MN7I6Z3A, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.542939249 (588h52m32.414022294s) [noise=true , address=45.91.116.9:4261 , piece_id=F5ZJ52QY2UZWOTQWKTBRSSACRRM4UBOMGJ3HIZKAWNTTBO3TFPQA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.543265000 (588h52m32.414348045s) [noise=true , address=77.90.31.21:4878 , piece_id=NCT377CAKGUOSA4TE2VZ5BUUH5P763ARAVCGLL3AZZWDM5264FHA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.689012514 (588h52m32.560095549s) [noise=true , address=45.91.116.11:4119 , piece_id=IKGHK5WI7SAAYUEMAMZVNNSIIQUCY2DY4XEP5WZZ56JOXC7AE6DA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:33.151625280 (588h52m33.022708315s) [noise=true , address=77.90.31.18:4805 , piece_id=JAHULSRJ4CQX2CVM7FRDYU42QP65MDYZMEDA67FAU5TC5XH63KSA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:33.151646832 (588h52m33.022729867s) [seg_pos={0 0}, upload=516232]: Segment reached optimal threshold of 13 pieces. Dec 30 04:21:33.151829930 (588h52m33.022912975s) [noise=true , address=45.91.116.14:4190 , piece_id=GNHZLIUFUWIU75NATSTLAZYPCOADGS5SQOG7PDWYOFIEAHKLA7RQ, seg_pos={0 0}, upload=516232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:48016->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:48016->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48016->45.91.116.14:4190: use of closed network connection Dec 30 04:21:33.151835961 (588h52m33.022919006s) [noise=true , address=45.91.116.14:4190 , piece_id=GNHZLIUFUWIU75NATSTLAZYPCOADGS5SQOG7PDWYOFIEAHKLA7RQ, seg_pos={0 0}, upload=516232]: Long tail cancellation due to optimal threshold Dec 30 04:21:33.152139480 (588h52m33.023222515s) [noise=true , address=77.90.31.13:4626 , piece_id=7JQWFFWMUTOQC6UOPEY33UW6CT6VB7KYSX7OJOJLHAB6MMLDDFJQ, seg_pos={0 0}, upload=516232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 30 04:21:33.152158535 (588h52m33.02324157s) [noise=true , address=77.90.31.13:4626 , piece_id=7JQWFFWMUTOQC6UOPEY33UW6CT6VB7KYSX7OJOJLHAB6MMLDDFJQ, seg_pos={0 0}, upload=516232]: Long tail cancellation due to optimal threshold Dec 30 04:21:33.152167142 (588h52m33.023250188s) [noise=true , address=45.91.116.10:4081 , piece_id=2EHXRG6VDQRWBYNMTTFZKZVRC5WQSTFQ2YS6KDGTAKRN3WYJV5WA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 30 04:21:33.152177231 (588h52m33.023260266s) [noise=true , address=45.91.116.10:4081 , piece_id=2EHXRG6VDQRWBYNMTTFZKZVRC5WQSTFQ2YS6KDGTAKRN3WYJV5WA, seg_pos={0 0}, upload=516232]: Long tail cancellation due to optimal threshold Dec 30 04:21:33.152199805 (588h52m33.02328285s) [seg_pos={0 0}, upload=516232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:21:33.152290857 (588h52m33.023373893s) [upload=516232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:21:33.152297861 (588h52m33.023380906s) [upload=516232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:21:33.448498170 (588h52m33.319581205s) [upload=516232]: Done uploading segments... Dec 30 04:21:33.448521505 (588h52m33.31960454s) [upload=516232]: Upload finished. err: <nil> Duration: 6.023935339s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XX...XXXX...XXXX...X.X...XX..X.X..XX.......XX..X.X...X.X..X] duration: 30.003489971s trace: https://jaeger.dev.storj.io/trace/455d6de5426ec2ab FAIL (2025-12-30 04:21:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 04:21:30.007567269 (588h52m29.878650314s) [upload=516232]: Starting upload Dec 30 04:21:30.007641130 (588h52m29.878724165s) [upload=516232]: Done starting upload Dec 30 04:21:30.007779433 (588h52m29.878862468s) [upload=516232]: Uploading segments... Dec 30 04:21:30.007885655 (588h52m29.8789687s) [upload=516232]: Got next segment. Inline: false Dec 30 04:21:30.007929128 (588h52m29.879012183s) [upload=516232]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:21:30.007934989 (588h52m29.879018024s) [upload=516232]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:21:30.253393762 (588h52m30.124476798s) [seg_pos={0 0}, upload=516232]: Begin upload segment... Dec 30 04:21:30.253440952 (588h52m30.124523997s) [seg_pos={0 0}, upload=516232]: Done begin upload segment. Dec 30 04:21:30.253477732 (588h52m30.124560797s) [noise=true , address=145.79.227.7:4293 , piece_id=6D2Y5RNCDDQZ5NZGVVAMY64NDMXAC5G7ZQBN2GI6LJUMRLSKLPYA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.253902430 (588h52m30.124985476s) [noise=true , address=145.79.227.20:4439 , piece_id=P6N5ZPK3CXFBUL4J6H6ZRUQ3Y322XNDTX6UQZOVQYFYKKH433E2Q, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.254328202 (588h52m30.125411237s) [noise=true , address=45.91.116.9:4261 , piece_id=F5ZJ52QY2UZWOTQWKTBRSSACRRM4UBOMGJ3HIZKAWNTTBO3TFPQA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.254869542 (588h52m30.125952577s) [noise=true , address=45.91.116.10:4081 , piece_id=2EHXRG6VDQRWBYNMTTFZKZVRC5WQSTFQ2YS6KDGTAKRN3WYJV5WA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.255276017 (588h52m30.126359052s) [noise=true , address=145.79.227.5:4222 , piece_id=SARA7UO4QZ7DZEKROQLDYBUQVHOWEEWQUMFECMF3CXJG3K5CUXEA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.255392348 (588h52m30.126475383s) [noise=true , address=77.90.31.13:4626 , piece_id=7JQWFFWMUTOQC6UOPEY33UW6CT6VB7KYSX7OJOJLHAB6MMLDDFJQ, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.255652634 (588h52m30.126735669s) [noise=true , address=77.90.31.18:4805 , piece_id=JAHULSRJ4CQX2CVM7FRDYU42QP65MDYZMEDA67FAU5TC5XH63KSA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.255886740 (588h52m30.126969785s) [noise=true , address=45.91.116.13:4155 , piece_id=CE7Z2YRR33ISP2FVWKNZ7NG2NK3JTFANPWFNPRXYRKX7MN7I6Z3A, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.256068315 (588h52m30.127151351s) [noise=true , address=145.79.227.25:4619 , piece_id=VQIXL5MALTMYE6ZTR5ZT5XPS6MJTVYEASB4B7ARADAVASWBODW7Q, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.256243800 (588h52m30.127326835s) [noise=true , address=145.79.227.22:4509 , piece_id=NGXZOJLBEEJLJLAGBPS2UPWELZKMQMYQB2RKKGUTOSK5F4X6AKOA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.256475631 (588h52m30.127558666s) [noise=true , address=77.90.31.21:4878 , piece_id=NCT377CAKGUOSA4TE2VZ5BUUH5P763ARAVCGLL3AZZWDM5264FHA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.256643480 (588h52m30.127726515s) [noise=true , address=145.79.227.36:4517 , piece_id=WX3UKKPRD6TXM5C3RT66LT34VQILJTMXPUVMRLLU2MMNCJV4HPXA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.256923003 (588h52m30.128006048s) [noise=true , address=145.79.227.11:4730 , piece_id=DN4OPI2FP7K2A4XNQPT7U6A5GZ6WQXRLHN6R5JKB7WQITTSJNZAA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.257027132 (588h52m30.128110177s) [noise=true , address=45.91.116.14:4190 , piece_id=GNHZLIUFUWIU75NATSTLAZYPCOADGS5SQOG7PDWYOFIEAHKLA7RQ, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.257374593 (588h52m30.128457638s) [noise=true , address=45.91.116.11:4119 , piece_id=IKGHK5WI7SAAYUEMAMZVNNSIIQUCY2DY4XEP5WZZ56JOXC7AE6DA, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:30.261062376 (588h52m30.132145421s) [noise=true , address=145.79.227.9:4367 , piece_id=MBVV7CQIKR6RYELQ7DYI42BJ75A5DHYXOHZSQENA5YTDHEI5TXQQ, seg_pos={0 0}, upload=516232]: Uploading piece... Dec 30 04:21:31.870217596 (588h52m31.741300641s) [upload=516232]: Next returned nil segment Dec 30 04:21:31.870375816 (588h52m31.741458872s) [upload=516232]: Waiting for error group managing segments... Dec 30 04:21:32.250613077 (588h52m32.121696122s) [noise=true , address=145.79.227.22:4509 , piece_id=NGXZOJLBEEJLJLAGBPS2UPWELZKMQMYQB2RKKGUTOSK5F4X6AKOA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.252727573 (588h52m32.123810618s) [noise=true , address=145.79.227.36:4517 , piece_id=WX3UKKPRD6TXM5C3RT66LT34VQILJTMXPUVMRLLU2MMNCJV4HPXA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.261585852 (588h52m32.132668887s) [noise=true , address=145.79.227.20:4439 , piece_id=P6N5ZPK3CXFBUL4J6H6ZRUQ3Y322XNDTX6UQZOVQYFYKKH433E2Q, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.263402672 (588h52m32.134485717s) [noise=true , address=145.79.227.7:4293 , piece_id=6D2Y5RNCDDQZ5NZGVVAMY64NDMXAC5G7ZQBN2GI6LJUMRLSKLPYA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.266322282 (588h52m32.137405317s) [noise=true , address=145.79.227.25:4619 , piece_id=VQIXL5MALTMYE6ZTR5ZT5XPS6MJTVYEASB4B7ARADAVASWBODW7Q, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.267473644 (588h52m32.138556689s) [noise=true , address=145.79.227.5:4222 , piece_id=SARA7UO4QZ7DZEKROQLDYBUQVHOWEEWQUMFECMF3CXJG3K5CUXEA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.271410441 (588h52m32.142493496s) [noise=true , address=145.79.227.11:4730 , piece_id=DN4OPI2FP7K2A4XNQPT7U6A5GZ6WQXRLHN6R5JKB7WQITTSJNZAA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.286605166 (588h52m32.157688201s) [noise=true , address=145.79.227.9:4367 , piece_id=MBVV7CQIKR6RYELQ7DYI42BJ75A5DHYXOHZSQENA5YTDHEI5TXQQ, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.508009643 (588h52m32.379092678s) [noise=true , address=45.91.116.13:4155 , piece_id=CE7Z2YRR33ISP2FVWKNZ7NG2NK3JTFANPWFNPRXYRKX7MN7I6Z3A, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.542939249 (588h52m32.414022294s) [noise=true , address=45.91.116.9:4261 , piece_id=F5ZJ52QY2UZWOTQWKTBRSSACRRM4UBOMGJ3HIZKAWNTTBO3TFPQA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.543265000 (588h52m32.414348045s) [noise=true , address=77.90.31.21:4878 , piece_id=NCT377CAKGUOSA4TE2VZ5BUUH5P763ARAVCGLL3AZZWDM5264FHA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:32.689012514 (588h52m32.560095549s) [noise=true , address=45.91.116.11:4119 , piece_id=IKGHK5WI7SAAYUEMAMZVNNSIIQUCY2DY4XEP5WZZ56JOXC7AE6DA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:33.151625280 (588h52m33.022708315s) [noise=true , address=77.90.31.18:4805 , piece_id=JAHULSRJ4CQX2CVM7FRDYU42QP65MDYZMEDA67FAU5TC5XH63KSA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: <nil> Dec 30 04:21:33.151646832 (588h52m33.022729867s) [seg_pos={0 0}, upload=516232]: Segment reached optimal threshold of 13 pieces. Dec 30 04:21:33.151829930 (588h52m33.022912975s) [noise=true , address=45.91.116.14:4190 , piece_id=GNHZLIUFUWIU75NATSTLAZYPCOADGS5SQOG7PDWYOFIEAHKLA7RQ, seg_pos={0 0}, upload=516232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:48016->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:48016->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48016->45.91.116.14:4190: use of closed network connection Dec 30 04:21:33.151835961 (588h52m33.022919006s) [noise=true , address=45.91.116.14:4190 , piece_id=GNHZLIUFUWIU75NATSTLAZYPCOADGS5SQOG7PDWYOFIEAHKLA7RQ, seg_pos={0 0}, upload=516232]: Long tail cancellation due to optimal threshold Dec 30 04:21:33.152139480 (588h52m33.023222515s) [noise=true , address=77.90.31.13:4626 , piece_id=7JQWFFWMUTOQC6UOPEY33UW6CT6VB7KYSX7OJOJLHAB6MMLDDFJQ, seg_pos={0 0}, upload=516232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 30 04:21:33.152158535 (588h52m33.02324157s) [noise=true , address=77.90.31.13:4626 , piece_id=7JQWFFWMUTOQC6UOPEY33UW6CT6VB7KYSX7OJOJLHAB6MMLDDFJQ, seg_pos={0 0}, upload=516232]: Long tail cancellation due to optimal threshold Dec 30 04:21:33.152167142 (588h52m33.023250188s) [noise=true , address=45.91.116.10:4081 , piece_id=2EHXRG6VDQRWBYNMTTFZKZVRC5WQSTFQ2YS6KDGTAKRN3WYJV5WA, seg_pos={0 0}, upload=516232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 30 04:21:33.152177231 (588h52m33.023260266s) [noise=true , address=45.91.116.10:4081 , piece_id=2EHXRG6VDQRWBYNMTTFZKZVRC5WQSTFQ2YS6KDGTAKRN3WYJV5WA, seg_pos={0 0}, upload=516232]: Long tail cancellation due to optimal threshold Dec 30 04:21:33.152199805 (588h52m33.02328285s) [seg_pos={0 0}, upload=516232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:21:33.152290857 (588h52m33.023373893s) [upload=516232]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:21:33.152297861 (588h52m33.023380906s) [upload=516232]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:21:33.448498170 (588h52m33.319581205s) [upload=516232]: Done uploading segments... Dec 30 04:21:33.448521505 (588h52m33.31960454s) [upload=516232]: Upload finished. err: <nil> Duration: 30.003513917s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.................X..XXX............XX..X.....X..............] duration: 3.505315169s trace: https://jaeger.dev.storj.io/trace/408434392b868da5 FAIL (2025-12-30 04:21:47 UTC) orbiter warning: context deadline exceeded Duration: 3.505339876s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X............X........................] duration: 39.945596887s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:21:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 39.94561909s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.XX...XXXX...XXXX...X.X...XX..X.X..XX.......XX..X.X...X.X..] duration: 7.003347486s trace: https://jaeger.dev.storj.io/trace/5ddfb7703d21fe6b FAIL (2025-12-30 04:21:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: VGKTEMP3MWZ55HQN7IEHRAYAAVLTBQ52LVM34LQTX7SUHKXEDGEA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: SA3EJEFWSOBSA2H5DY6MIA4HNPZRCUHY3CHCUZERGTOCMOYRT3SQ) context deadline exceeded ======LOGS====== 30 04:19:55.135140946 (588h50m55.006223991s) [upload=516206]: Starting upload Dec 30 04:19:55.135231428 (588h50m55.006314473s) [upload=516206]: Done starting upload Dec 30 04:19:55.135320558 (588h50m55.006403593s) [upload=516206]: Uploading segments... Dec 30 04:19:55.135626240 (588h50m55.006709276s) [upload=516206]: Got next segment. Inline: false Dec 30 04:19:55.135673320 (588h50m55.006756355s) [upload=516206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:19:55.135678750 (588h50m55.006761795s) [upload=516206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:19:55.381264095 (588h50m55.25234714s) [seg_pos={0 0}, upload=516206]: Begin upload segment... Dec 30 04:19:55.381330651 (588h50m55.252413686s) [seg_pos={0 0}, upload=516206]: Done begin upload segment. Dec 30 04:19:55.381355699 (588h50m55.252438734s) [noise=true , address=45.91.116.6:4976 , piece_id=XEAC3D3OVBM2GBATOWZQYCK7X5OKBWXDJ5JV37MIYPWGJWGH5PTQ, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.381456791 (588h50m55.252539836s) [noise=true , address=145.79.227.7:4295 , piece_id=XNYRW3QAFPS4YVESDS5CGA7LWBJVGJG7O67ITXL7BK56MQCH4VGQ, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.381897320 (588h50m55.252980356s) [noise=true , address=145.79.227.39:4013 , piece_id=YNLWQNYR4T4DVUQFAE7FXJSIL3UXIDJESXE32A6OTZFHF5QVLLQA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.382463739 (588h50m55.253546774s) [noise=true , address=145.79.227.41:4086 , piece_id=W2EPQURAF4PQF4T5LM657GSA4ZLWPQIKBHXPDQUD6VS4RRUPGGVA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.382826791 (588h50m55.253909826s) [noise=true , address=145.79.227.48:4301 , piece_id=D22NAUK7B6C3LHLNFF4PZAYPSN2OF5MPXKG2XL6T7WZZ2PKUWT4Q, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.383246430 (588h50m55.254329465s) [noise=true , address=145.79.227.42:4123 , piece_id=7QYTJ2Y5H63C3FFHQKXLPUH6AD2CIK4XVTEBBH3JJXQT4G3PYCQA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.383583361 (588h50m55.254666396s) [noise=true , address=45.91.116.13:4154 , piece_id=HRFJAHJUSC3VDDYVTSAS6TE4WIKQLQ4IADFSLZNEI7XTAHXIXYFA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.383891708 (588h50m55.254974753s) [noise=true , address=145.79.227.52:4445 , piece_id=BVBRSZJY6RYQYKCGUIRTMLFFEHOUSV3JGZX5EFSDBC2GYWNSSOUA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.384226646 (588h50m55.255309691s) [noise=true , address=77.90.31.17:4770 , piece_id=DSV5NRGK7GX7JRKKP52WGJCUW7447TEXP57R4GFNEZ6744KCRRJA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.385316631 (588h50m55.256399676s) [noise=true , address=145.79.227.17:4971 , piece_id=QGANMQTYZYUXXLCNJSCJKC75QN2PJW6MX2XKW4GWGQFVBV3PIPEA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.386735603 (588h50m55.257818638s) [noise=true , address=145.79.227.19:4077 , piece_id=GIEMI44XUDU5D5CUAWIYI5RZYM2S4SJIZ6SYE245CSGUSTQDMRZQ, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.386834572 (588h50m55.257917607s) [noise=true , address=45.91.116.11:4118 , piece_id=VGKTEMP3MWZ55HQN7IEHRAYAAVLTBQ52LVM34LQTX7SUHKXEDGEA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.387980334 (588h50m55.259063369s) [noise=true , address=145.79.227.34:4905 , piece_id=2OX5P2CK6YGDZERLBMNKQ3KTPAYQ2H3RKFIHYBWWLN5R7POGXIKA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.389033550 (588h50m55.260116585s) [noise=true , address=145.79.227.26:4655 , piece_id=SXDGYLTY4OAWO7JBA4EFYI2LNENGWD6KUJAH5F76APOZTQWRAR2Q, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.391714836 (588h50m55.262797911s) [noise=true , address=77.90.31.12:4590 , piece_id=SA3EJEFWSOBSA2H5DY6MIA4HNPZRCUHY3CHCUZERGTOCMOYRT3SQ, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.393291809 (588h50m55.264374854s) [noise=true , address=77.90.31.19:4844 , piece_id=QYRA6RNKTAEOTT2D6CCKU5AWPIH3F57TURSP5JN7NM5DVKV6QE3A, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:56.987656873 (588h50m56.858739928s) [upload=516206]: Next returned nil segment Dec 30 04:19:56.987675068 (588h50m56.858758113s) [upload=516206]: Waiting for error group managing segments... Dec 30 04:19:57.372607008 (588h50m57.243690053s) [noise=true , address=145.79.227.19:4077 , piece_id=GIEMI44XUDU5D5CUAWIYI5RZYM2S4SJIZ6SYE245CSGUSTQDMRZQ, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.376569363 (588h50m57.247652399s) [noise=true , address=145.79.227.39:4013 , piece_id=YNLWQNYR4T4DVUQFAE7FXJSIL3UXIDJESXE32A6OTZFHF5QVLLQA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.377523400 (588h50m57.248606445s) [noise=true , address=145.79.227.7:4295 , piece_id=XNYRW3QAFPS4YVESDS5CGA7LWBJVGJG7O67ITXL7BK56MQCH4VGQ, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.378723656 (588h50m57.249806691s) [noise=true , address=145.79.227.42:4123 , piece_id=7QYTJ2Y5H63C3FFHQKXLPUH6AD2CIK4XVTEBBH3JJXQT4G3PYCQA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.379527196 (588h50m57.250610241s) [noise=true , address=145.79.227.52:4445 , piece_id=BVBRSZJY6RYQYKCGUIRTMLFFEHOUSV3JGZX5EFSDBC2GYWNSSOUA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.379902641 (588h50m57.250985696s) [noise=true , address=145.79.227.48:4301 , piece_id=D22NAUK7B6C3LHLNFF4PZAYPSN2OF5MPXKG2XL6T7WZZ2PKUWT4Q, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.390897157 (588h50m57.261980193s) [noise=true , address=145.79.227.17:4971 , piece_id=QGANMQTYZYUXXLCNJSCJKC75QN2PJW6MX2XKW4GWGQFVBV3PIPEA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.391859931 (588h50m57.262942966s) [noise=true , address=145.79.227.41:4086 , piece_id=W2EPQURAF4PQF4T5LM657GSA4ZLWPQIKBHXPDQUD6VS4RRUPGGVA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.421983235 (588h50m57.29306628s) [noise=true , address=145.79.227.26:4655 , piece_id=SXDGYLTY4OAWO7JBA4EFYI2LNENGWD6KUJAH5F76APOZTQWRAR2Q, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.570507417 (588h50m57.441590452s) [noise=true , address=77.90.31.17:4770 , piece_id=DSV5NRGK7GX7JRKKP52WGJCUW7447TEXP57R4GFNEZ6744KCRRJA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.673837066 (588h50m57.544920111s) [noise=true , address=77.90.31.12:4590 , piece_id=SA3EJEFWSOBSA2H5DY6MIA4HNPZRCUHY3CHCUZERGTOCMOYRT3SQ, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.675113557 (588h50m57.546196592s) [noise=true , address=45.91.116.11:4118 , piece_id=VGKTEMP3MWZ55HQN7IEHRAYAAVLTBQ52LVM34LQTX7SUHKXEDGEA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.724496456 (588h50m57.595579501s) [noise=true , address=45.91.116.13:4154 , piece_id=HRFJAHJUSC3VDDYVTSAS6TE4WIKQLQ4IADFSLZNEI7XTAHXIXYFA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.724522285 (588h50m57.59560532s) [seg_pos={0 0}, upload=516206]: Segment reached optimal threshold of 13 pieces. Dec 30 04:19:57.724753416 (588h50m57.595836461s) [noise=true , address=77.90.31.19:4844 , piece_id=QYRA6RNKTAEOTT2D6CCKU5AWPIH3F57TURSP5JN7NM5DVKV6QE3A, seg_pos={0 0}, upload=516206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:55176->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:55176->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55176->77.90.31.19:4844: use of closed network connection Dec 30 04:19:57.724760900 (588h50m57.595843945s) [noise=true , address=77.90.31.19:4844 , piece_id=QYRA6RNKTAEOTT2D6CCKU5AWPIH3F57TURSP5JN7NM5DVKV6QE3A, seg_pos={0 0}, upload=516206]: Long tail cancellation due to optimal threshold Dec 30 04:19:57.724964888 (588h50m57.596047933s) [noise=true , address=45.91.116.6:4976 , piece_id=XEAC3D3OVBM2GBATOWZQYCK7X5OKBWXDJ5JV37MIYPWGJWGH5PTQ, seg_pos={0 0}, upload=516206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 30 04:19:57.725004894 (588h50m57.596087939s) [noise=true , address=45.91.116.6:4976 , piece_id=XEAC3D3OVBM2GBATOWZQYCK7X5OKBWXDJ5JV37MIYPWGJWGH5PTQ, seg_pos={0 0}, upload=516206]: Long tail cancellation due to optimal threshold Dec 30 04:19:57.725157575 (588h50m57.59624062s) [noise=true , address=145.79.227.34:4905 , piece_id=2OX5P2CK6YGDZERLBMNKQ3KTPAYQ2H3RKFIHYBWWLN5R7POGXIKA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 04:19:57.725166743 (588h50m57.596249788s) [noise=true , address=145.79.227.34:4905 , piece_id=2OX5P2CK6YGDZERLBMNKQ3KTPAYQ2H3RKFIHYBWWLN5R7POGXIKA, seg_pos={0 0}, upload=516206]: Long tail cancellation due to optimal threshold Dec 30 04:19:57.725187592 (588h50m57.596270637s) [seg_pos={0 0}, upload=516206]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:19:57.725274238 (588h50m57.596357273s) [upload=516206]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:19:57.725280099 (588h50m57.596363134s) [upload=516206]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:19:58.018827876 (588h50m57.889910922s) [upload=516206]: Done uploading segments... Dec 30 04:19:58.018852063 (588h50m57.889935098s) [upload=516206]: Upload finished. err: <nil> Duration: 7.003377804s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..................X..XXX............XX..X.....X.............] duration: 3.588295226s trace: https://jaeger.dev.storj.io/trace/408434392b868da5 FAIL (2025-12-30 04:21:02 UTC) orbiter warning: context deadline exceeded Duration: 3.588438237s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XX.XX...XXXX...XXXX...X.X...XX..X.X..XX.......XX..X.X...X.X.] duration: 5.956310959s trace: https://jaeger.dev.storj.io/trace/5ddfb7703d21fe6b FAIL (2025-12-30 04:20:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: VGKTEMP3MWZ55HQN7IEHRAYAAVLTBQ52LVM34LQTX7SUHKXEDGEA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: SA3EJEFWSOBSA2H5DY6MIA4HNPZRCUHY3CHCUZERGTOCMOYRT3SQ) context deadline exceeded ======LOGS====== 30 04:19:55.135140946 (588h50m55.006223991s) [upload=516206]: Starting upload Dec 30 04:19:55.135231428 (588h50m55.006314473s) [upload=516206]: Done starting upload Dec 30 04:19:55.135320558 (588h50m55.006403593s) [upload=516206]: Uploading segments... Dec 30 04:19:55.135626240 (588h50m55.006709276s) [upload=516206]: Got next segment. Inline: false Dec 30 04:19:55.135673320 (588h50m55.006756355s) [upload=516206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:19:55.135678750 (588h50m55.006761795s) [upload=516206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:19:55.381264095 (588h50m55.25234714s) [seg_pos={0 0}, upload=516206]: Begin upload segment... Dec 30 04:19:55.381330651 (588h50m55.252413686s) [seg_pos={0 0}, upload=516206]: Done begin upload segment. Dec 30 04:19:55.381355699 (588h50m55.252438734s) [noise=true , address=45.91.116.6:4976 , piece_id=XEAC3D3OVBM2GBATOWZQYCK7X5OKBWXDJ5JV37MIYPWGJWGH5PTQ, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.381456791 (588h50m55.252539836s) [noise=true , address=145.79.227.7:4295 , piece_id=XNYRW3QAFPS4YVESDS5CGA7LWBJVGJG7O67ITXL7BK56MQCH4VGQ, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.381897320 (588h50m55.252980356s) [noise=true , address=145.79.227.39:4013 , piece_id=YNLWQNYR4T4DVUQFAE7FXJSIL3UXIDJESXE32A6OTZFHF5QVLLQA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.382463739 (588h50m55.253546774s) [noise=true , address=145.79.227.41:4086 , piece_id=W2EPQURAF4PQF4T5LM657GSA4ZLWPQIKBHXPDQUD6VS4RRUPGGVA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.382826791 (588h50m55.253909826s) [noise=true , address=145.79.227.48:4301 , piece_id=D22NAUK7B6C3LHLNFF4PZAYPSN2OF5MPXKG2XL6T7WZZ2PKUWT4Q, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.383246430 (588h50m55.254329465s) [noise=true , address=145.79.227.42:4123 , piece_id=7QYTJ2Y5H63C3FFHQKXLPUH6AD2CIK4XVTEBBH3JJXQT4G3PYCQA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.383583361 (588h50m55.254666396s) [noise=true , address=45.91.116.13:4154 , piece_id=HRFJAHJUSC3VDDYVTSAS6TE4WIKQLQ4IADFSLZNEI7XTAHXIXYFA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.383891708 (588h50m55.254974753s) [noise=true , address=145.79.227.52:4445 , piece_id=BVBRSZJY6RYQYKCGUIRTMLFFEHOUSV3JGZX5EFSDBC2GYWNSSOUA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.384226646 (588h50m55.255309691s) [noise=true , address=77.90.31.17:4770 , piece_id=DSV5NRGK7GX7JRKKP52WGJCUW7447TEXP57R4GFNEZ6744KCRRJA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.385316631 (588h50m55.256399676s) [noise=true , address=145.79.227.17:4971 , piece_id=QGANMQTYZYUXXLCNJSCJKC75QN2PJW6MX2XKW4GWGQFVBV3PIPEA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.386735603 (588h50m55.257818638s) [noise=true , address=145.79.227.19:4077 , piece_id=GIEMI44XUDU5D5CUAWIYI5RZYM2S4SJIZ6SYE245CSGUSTQDMRZQ, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.386834572 (588h50m55.257917607s) [noise=true , address=45.91.116.11:4118 , piece_id=VGKTEMP3MWZ55HQN7IEHRAYAAVLTBQ52LVM34LQTX7SUHKXEDGEA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.387980334 (588h50m55.259063369s) [noise=true , address=145.79.227.34:4905 , piece_id=2OX5P2CK6YGDZERLBMNKQ3KTPAYQ2H3RKFIHYBWWLN5R7POGXIKA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.389033550 (588h50m55.260116585s) [noise=true , address=145.79.227.26:4655 , piece_id=SXDGYLTY4OAWO7JBA4EFYI2LNENGWD6KUJAH5F76APOZTQWRAR2Q, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.391714836 (588h50m55.262797911s) [noise=true , address=77.90.31.12:4590 , piece_id=SA3EJEFWSOBSA2H5DY6MIA4HNPZRCUHY3CHCUZERGTOCMOYRT3SQ, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.393291809 (588h50m55.264374854s) [noise=true , address=77.90.31.19:4844 , piece_id=QYRA6RNKTAEOTT2D6CCKU5AWPIH3F57TURSP5JN7NM5DVKV6QE3A, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:56.987656873 (588h50m56.858739928s) [upload=516206]: Next returned nil segment Dec 30 04:19:56.987675068 (588h50m56.858758113s) [upload=516206]: Waiting for error group managing segments... Dec 30 04:19:57.372607008 (588h50m57.243690053s) [noise=true , address=145.79.227.19:4077 , piece_id=GIEMI44XUDU5D5CUAWIYI5RZYM2S4SJIZ6SYE245CSGUSTQDMRZQ, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.376569363 (588h50m57.247652399s) [noise=true , address=145.79.227.39:4013 , piece_id=YNLWQNYR4T4DVUQFAE7FXJSIL3UXIDJESXE32A6OTZFHF5QVLLQA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.377523400 (588h50m57.248606445s) [noise=true , address=145.79.227.7:4295 , piece_id=XNYRW3QAFPS4YVESDS5CGA7LWBJVGJG7O67ITXL7BK56MQCH4VGQ, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.378723656 (588h50m57.249806691s) [noise=true , address=145.79.227.42:4123 , piece_id=7QYTJ2Y5H63C3FFHQKXLPUH6AD2CIK4XVTEBBH3JJXQT4G3PYCQA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.379527196 (588h50m57.250610241s) [noise=true , address=145.79.227.52:4445 , piece_id=BVBRSZJY6RYQYKCGUIRTMLFFEHOUSV3JGZX5EFSDBC2GYWNSSOUA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.379902641 (588h50m57.250985696s) [noise=true , address=145.79.227.48:4301 , piece_id=D22NAUK7B6C3LHLNFF4PZAYPSN2OF5MPXKG2XL6T7WZZ2PKUWT4Q, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.390897157 (588h50m57.261980193s) [noise=true , address=145.79.227.17:4971 , piece_id=QGANMQTYZYUXXLCNJSCJKC75QN2PJW6MX2XKW4GWGQFVBV3PIPEA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.391859931 (588h50m57.262942966s) [noise=true , address=145.79.227.41:4086 , piece_id=W2EPQURAF4PQF4T5LM657GSA4ZLWPQIKBHXPDQUD6VS4RRUPGGVA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.421983235 (588h50m57.29306628s) [noise=true , address=145.79.227.26:4655 , piece_id=SXDGYLTY4OAWO7JBA4EFYI2LNENGWD6KUJAH5F76APOZTQWRAR2Q, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.570507417 (588h50m57.441590452s) [noise=true , address=77.90.31.17:4770 , piece_id=DSV5NRGK7GX7JRKKP52WGJCUW7447TEXP57R4GFNEZ6744KCRRJA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.673837066 (588h50m57.544920111s) [noise=true , address=77.90.31.12:4590 , piece_id=SA3EJEFWSOBSA2H5DY6MIA4HNPZRCUHY3CHCUZERGTOCMOYRT3SQ, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.675113557 (588h50m57.546196592s) [noise=true , address=45.91.116.11:4118 , piece_id=VGKTEMP3MWZ55HQN7IEHRAYAAVLTBQ52LVM34LQTX7SUHKXEDGEA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.724496456 (588h50m57.595579501s) [noise=true , address=45.91.116.13:4154 , piece_id=HRFJAHJUSC3VDDYVTSAS6TE4WIKQLQ4IADFSLZNEI7XTAHXIXYFA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.724522285 (588h50m57.59560532s) [seg_pos={0 0}, upload=516206]: Segment reached optimal threshold of 13 pieces. Dec 30 04:19:57.724753416 (588h50m57.595836461s) [noise=true , address=77.90.31.19:4844 , piece_id=QYRA6RNKTAEOTT2D6CCKU5AWPIH3F57TURSP5JN7NM5DVKV6QE3A, seg_pos={0 0}, upload=516206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:55176->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:55176->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55176->77.90.31.19:4844: use of closed network connection Dec 30 04:19:57.724760900 (588h50m57.595843945s) [noise=true , address=77.90.31.19:4844 , piece_id=QYRA6RNKTAEOTT2D6CCKU5AWPIH3F57TURSP5JN7NM5DVKV6QE3A, seg_pos={0 0}, upload=516206]: Long tail cancellation due to optimal threshold Dec 30 04:19:57.724964888 (588h50m57.596047933s) [noise=true , address=45.91.116.6:4976 , piece_id=XEAC3D3OVBM2GBATOWZQYCK7X5OKBWXDJ5JV37MIYPWGJWGH5PTQ, seg_pos={0 0}, upload=516206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 30 04:19:57.725004894 (588h50m57.596087939s) [noise=true , address=45.91.116.6:4976 , piece_id=XEAC3D3OVBM2GBATOWZQYCK7X5OKBWXDJ5JV37MIYPWGJWGH5PTQ, seg_pos={0 0}, upload=516206]: Long tail cancellation due to optimal threshold Dec 30 04:19:57.725157575 (588h50m57.59624062s) [noise=true , address=145.79.227.34:4905 , piece_id=2OX5P2CK6YGDZERLBMNKQ3KTPAYQ2H3RKFIHYBWWLN5R7POGXIKA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 04:19:57.725166743 (588h50m57.596249788s) [noise=true , address=145.79.227.34:4905 , piece_id=2OX5P2CK6YGDZERLBMNKQ3KTPAYQ2H3RKFIHYBWWLN5R7POGXIKA, seg_pos={0 0}, upload=516206]: Long tail cancellation due to optimal threshold Dec 30 04:19:57.725187592 (588h50m57.596270637s) [seg_pos={0 0}, upload=516206]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:19:57.725274238 (588h50m57.596357273s) [upload=516206]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:19:57.725280099 (588h50m57.596363134s) [upload=516206]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:19:58.018827876 (588h50m57.889910922s) [upload=516206]: Done uploading segments... Dec 30 04:19:58.018852063 (588h50m57.889935098s) [upload=516206]: Upload finished. err: <nil> Duration: 5.95637441s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X............X.......................] duration: 41.564725705s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:20:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 41.564741926s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.XX.XX...XXXX...XXXX...X.X...XX..X.X..XX.......XX..X.X...X.X] duration: 30.006154468s trace: https://jaeger.dev.storj.io/trace/5ddfb7703d21fe6b FAIL (2025-12-30 04:20:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: VGKTEMP3MWZ55HQN7IEHRAYAAVLTBQ52LVM34LQTX7SUHKXEDGEA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: SA3EJEFWSOBSA2H5DY6MIA4HNPZRCUHY3CHCUZERGTOCMOYRT3SQ) context deadline exceeded ======LOGS====== 30 04:19:55.135140946 (588h50m55.006223991s) [upload=516206]: Starting upload Dec 30 04:19:55.135231428 (588h50m55.006314473s) [upload=516206]: Done starting upload Dec 30 04:19:55.135320558 (588h50m55.006403593s) [upload=516206]: Uploading segments... Dec 30 04:19:55.135626240 (588h50m55.006709276s) [upload=516206]: Got next segment. Inline: false Dec 30 04:19:55.135673320 (588h50m55.006756355s) [upload=516206]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:19:55.135678750 (588h50m55.006761795s) [upload=516206]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:19:55.381264095 (588h50m55.25234714s) [seg_pos={0 0}, upload=516206]: Begin upload segment... Dec 30 04:19:55.381330651 (588h50m55.252413686s) [seg_pos={0 0}, upload=516206]: Done begin upload segment. Dec 30 04:19:55.381355699 (588h50m55.252438734s) [noise=true , address=45.91.116.6:4976 , piece_id=XEAC3D3OVBM2GBATOWZQYCK7X5OKBWXDJ5JV37MIYPWGJWGH5PTQ, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.381456791 (588h50m55.252539836s) [noise=true , address=145.79.227.7:4295 , piece_id=XNYRW3QAFPS4YVESDS5CGA7LWBJVGJG7O67ITXL7BK56MQCH4VGQ, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.381897320 (588h50m55.252980356s) [noise=true , address=145.79.227.39:4013 , piece_id=YNLWQNYR4T4DVUQFAE7FXJSIL3UXIDJESXE32A6OTZFHF5QVLLQA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.382463739 (588h50m55.253546774s) [noise=true , address=145.79.227.41:4086 , piece_id=W2EPQURAF4PQF4T5LM657GSA4ZLWPQIKBHXPDQUD6VS4RRUPGGVA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.382826791 (588h50m55.253909826s) [noise=true , address=145.79.227.48:4301 , piece_id=D22NAUK7B6C3LHLNFF4PZAYPSN2OF5MPXKG2XL6T7WZZ2PKUWT4Q, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.383246430 (588h50m55.254329465s) [noise=true , address=145.79.227.42:4123 , piece_id=7QYTJ2Y5H63C3FFHQKXLPUH6AD2CIK4XVTEBBH3JJXQT4G3PYCQA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.383583361 (588h50m55.254666396s) [noise=true , address=45.91.116.13:4154 , piece_id=HRFJAHJUSC3VDDYVTSAS6TE4WIKQLQ4IADFSLZNEI7XTAHXIXYFA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.383891708 (588h50m55.254974753s) [noise=true , address=145.79.227.52:4445 , piece_id=BVBRSZJY6RYQYKCGUIRTMLFFEHOUSV3JGZX5EFSDBC2GYWNSSOUA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.384226646 (588h50m55.255309691s) [noise=true , address=77.90.31.17:4770 , piece_id=DSV5NRGK7GX7JRKKP52WGJCUW7447TEXP57R4GFNEZ6744KCRRJA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.385316631 (588h50m55.256399676s) [noise=true , address=145.79.227.17:4971 , piece_id=QGANMQTYZYUXXLCNJSCJKC75QN2PJW6MX2XKW4GWGQFVBV3PIPEA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.386735603 (588h50m55.257818638s) [noise=true , address=145.79.227.19:4077 , piece_id=GIEMI44XUDU5D5CUAWIYI5RZYM2S4SJIZ6SYE245CSGUSTQDMRZQ, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.386834572 (588h50m55.257917607s) [noise=true , address=45.91.116.11:4118 , piece_id=VGKTEMP3MWZ55HQN7IEHRAYAAVLTBQ52LVM34LQTX7SUHKXEDGEA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.387980334 (588h50m55.259063369s) [noise=true , address=145.79.227.34:4905 , piece_id=2OX5P2CK6YGDZERLBMNKQ3KTPAYQ2H3RKFIHYBWWLN5R7POGXIKA, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.389033550 (588h50m55.260116585s) [noise=true , address=145.79.227.26:4655 , piece_id=SXDGYLTY4OAWO7JBA4EFYI2LNENGWD6KUJAH5F76APOZTQWRAR2Q, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.391714836 (588h50m55.262797911s) [noise=true , address=77.90.31.12:4590 , piece_id=SA3EJEFWSOBSA2H5DY6MIA4HNPZRCUHY3CHCUZERGTOCMOYRT3SQ, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:55.393291809 (588h50m55.264374854s) [noise=true , address=77.90.31.19:4844 , piece_id=QYRA6RNKTAEOTT2D6CCKU5AWPIH3F57TURSP5JN7NM5DVKV6QE3A, seg_pos={0 0}, upload=516206]: Uploading piece... Dec 30 04:19:56.987656873 (588h50m56.858739928s) [upload=516206]: Next returned nil segment Dec 30 04:19:56.987675068 (588h50m56.858758113s) [upload=516206]: Waiting for error group managing segments... Dec 30 04:19:57.372607008 (588h50m57.243690053s) [noise=true , address=145.79.227.19:4077 , piece_id=GIEMI44XUDU5D5CUAWIYI5RZYM2S4SJIZ6SYE245CSGUSTQDMRZQ, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.376569363 (588h50m57.247652399s) [noise=true , address=145.79.227.39:4013 , piece_id=YNLWQNYR4T4DVUQFAE7FXJSIL3UXIDJESXE32A6OTZFHF5QVLLQA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.377523400 (588h50m57.248606445s) [noise=true , address=145.79.227.7:4295 , piece_id=XNYRW3QAFPS4YVESDS5CGA7LWBJVGJG7O67ITXL7BK56MQCH4VGQ, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.378723656 (588h50m57.249806691s) [noise=true , address=145.79.227.42:4123 , piece_id=7QYTJ2Y5H63C3FFHQKXLPUH6AD2CIK4XVTEBBH3JJXQT4G3PYCQA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.379527196 (588h50m57.250610241s) [noise=true , address=145.79.227.52:4445 , piece_id=BVBRSZJY6RYQYKCGUIRTMLFFEHOUSV3JGZX5EFSDBC2GYWNSSOUA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.379902641 (588h50m57.250985696s) [noise=true , address=145.79.227.48:4301 , piece_id=D22NAUK7B6C3LHLNFF4PZAYPSN2OF5MPXKG2XL6T7WZZ2PKUWT4Q, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.390897157 (588h50m57.261980193s) [noise=true , address=145.79.227.17:4971 , piece_id=QGANMQTYZYUXXLCNJSCJKC75QN2PJW6MX2XKW4GWGQFVBV3PIPEA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.391859931 (588h50m57.262942966s) [noise=true , address=145.79.227.41:4086 , piece_id=W2EPQURAF4PQF4T5LM657GSA4ZLWPQIKBHXPDQUD6VS4RRUPGGVA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.421983235 (588h50m57.29306628s) [noise=true , address=145.79.227.26:4655 , piece_id=SXDGYLTY4OAWO7JBA4EFYI2LNENGWD6KUJAH5F76APOZTQWRAR2Q, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.570507417 (588h50m57.441590452s) [noise=true , address=77.90.31.17:4770 , piece_id=DSV5NRGK7GX7JRKKP52WGJCUW7447TEXP57R4GFNEZ6744KCRRJA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.673837066 (588h50m57.544920111s) [noise=true , address=77.90.31.12:4590 , piece_id=SA3EJEFWSOBSA2H5DY6MIA4HNPZRCUHY3CHCUZERGTOCMOYRT3SQ, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.675113557 (588h50m57.546196592s) [noise=true , address=45.91.116.11:4118 , piece_id=VGKTEMP3MWZ55HQN7IEHRAYAAVLTBQ52LVM34LQTX7SUHKXEDGEA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.724496456 (588h50m57.595579501s) [noise=true , address=45.91.116.13:4154 , piece_id=HRFJAHJUSC3VDDYVTSAS6TE4WIKQLQ4IADFSLZNEI7XTAHXIXYFA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: <nil> Dec 30 04:19:57.724522285 (588h50m57.59560532s) [seg_pos={0 0}, upload=516206]: Segment reached optimal threshold of 13 pieces. Dec 30 04:19:57.724753416 (588h50m57.595836461s) [noise=true , address=77.90.31.19:4844 , piece_id=QYRA6RNKTAEOTT2D6CCKU5AWPIH3F57TURSP5JN7NM5DVKV6QE3A, seg_pos={0 0}, upload=516206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:55176->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:55176->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55176->77.90.31.19:4844: use of closed network connection Dec 30 04:19:57.724760900 (588h50m57.595843945s) [noise=true , address=77.90.31.19:4844 , piece_id=QYRA6RNKTAEOTT2D6CCKU5AWPIH3F57TURSP5JN7NM5DVKV6QE3A, seg_pos={0 0}, upload=516206]: Long tail cancellation due to optimal threshold Dec 30 04:19:57.724964888 (588h50m57.596047933s) [noise=true , address=45.91.116.6:4976 , piece_id=XEAC3D3OVBM2GBATOWZQYCK7X5OKBWXDJ5JV37MIYPWGJWGH5PTQ, seg_pos={0 0}, upload=516206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 30 04:19:57.725004894 (588h50m57.596087939s) [noise=true , address=45.91.116.6:4976 , piece_id=XEAC3D3OVBM2GBATOWZQYCK7X5OKBWXDJ5JV37MIYPWGJWGH5PTQ, seg_pos={0 0}, upload=516206]: Long tail cancellation due to optimal threshold Dec 30 04:19:57.725157575 (588h50m57.59624062s) [noise=true , address=145.79.227.34:4905 , piece_id=2OX5P2CK6YGDZERLBMNKQ3KTPAYQ2H3RKFIHYBWWLN5R7POGXIKA, seg_pos={0 0}, upload=516206]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 04:19:57.725166743 (588h50m57.596249788s) [noise=true , address=145.79.227.34:4905 , piece_id=2OX5P2CK6YGDZERLBMNKQ3KTPAYQ2H3RKFIHYBWWLN5R7POGXIKA, seg_pos={0 0}, upload=516206]: Long tail cancellation due to optimal threshold Dec 30 04:19:57.725187592 (588h50m57.596270637s) [seg_pos={0 0}, upload=516206]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:19:57.725274238 (588h50m57.596357273s) [upload=516206]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:19:57.725280099 (588h50m57.596363134s) [upload=516206]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:19:58.018827876 (588h50m57.889910922s) [upload=516206]: Done uploading segments... Dec 30 04:19:58.018852063 (588h50m57.889935098s) [upload=516206]: Upload finished. err: <nil> Duration: 30.006177551s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...................X..XXX............XX..X.....X............] duration: 3.630230882s trace: https://jaeger.dev.storj.io/trace/408434392b868da5 FAIL (2025-12-30 04:20:05 UTC) orbiter warning: context deadline exceeded Duration: 3.630560049s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X............X......................] duration: 48.387836782s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:19:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 48.387870025s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....................X..XXX............XX..X.....X...........] duration: 5.000520792s trace: https://jaeger.dev.storj.io/trace/408434392b868da5 FAIL (2025-12-30 04:19:46 UTC) orbiter warning: context deadline exceeded Duration: 5.000541953s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [..XX.XX...XXXX...XXXX...X.X...XX..X.X..XX.......XX..X.X...X.] duration: 5.922414524s trace: https://jaeger.dev.storj.io/trace/24d3b8beaf199c44 FAIL (2025-12-30 04:19:34 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 04:18:49.496068011 (588h49m49.367151056s) [upload=516191]: Starting upload Dec 30 04:18:49.496163933 (588h49m49.367246968s) [upload=516191]: Done starting upload Dec 30 04:18:49.496258503 (588h49m49.367341538s) [upload=516191]: Uploading segments... Dec 30 04:18:49.496534699 (588h49m49.367617744s) [upload=516191]: Got next segment. Inline: false Dec 30 04:18:49.496860750 (588h49m49.367943795s) [upload=516191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:18:49.496867834 (588h49m49.367950869s) [upload=516191]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:18:49.739924243 (588h49m49.611007288s) [seg_pos={0 0}, upload=516191]: Begin upload segment... Dec 30 04:18:49.739981903 (588h49m49.611064948s) [seg_pos={0 0}, upload=516191]: Done begin upload segment. Dec 30 04:18:49.740032219 (588h49m49.611115264s) [noise=true , address=77.90.31.11:4553 , piece_id=UC7ZZLTIGMALZA5LFVRJ6F6CCS4UKLDHKE2HGR3ZVSG64W6U5K2A, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.740037448 (588h49m49.611120483s) [noise=true , address=145.79.227.11:4731 , piece_id=PG43LBI62W6NGHO5B64JM5HBC7DCHO73BQGIURAB2KJTCTX73F3Q, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.740784341 (588h49m49.611867386s) [noise=true , address=77.90.31.18:4808 , piece_id=YUNEX6VTNVZJEQC573BCO253ZQRCTAA4Z2ZREVQOR26FKI2EQTTA, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.741248845 (588h49m49.61233188s) [noise=true , address=145.79.227.12:4767 , piece_id=MNSOU6E3LNQDRP35J54JQJW5EX5ED4IHU5W3ANRIERMQ6K6YW44Q, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.741577111 (588h49m49.612660156s) [noise=true , address=145.79.227.2:4115 , piece_id=JGVQEXNFDE7VLALIUASO5XYGG6Y7JGZ7ITJPLUUU6EBPDVV3BH5A, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.741943278 (588h49m49.613026323s) [noise=true , address=145.79.227.5:4223 , piece_id=LMEDBWW3UGMVXBBGQPJ5R66JRLMGYSJPMCCIBUZU5Y53SSXHJ3GA, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.742230354 (588h49m49.613313399s) [noise=true , address=145.79.227.42:4122 , piece_id=6MSZTZ3U573XBLDLJGHCUH5TEO2ARLU72CZROAJ7MY3EX4HFJUJA, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.742502933 (588h49m49.613585978s) [noise=true , address=145.79.227.43:4157 , piece_id=6C7WJTZOUCQYIOURVSBMUIRIOTUY5TAMOTCWQ5CXNPEOV7CONDHA, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.742645615 (588h49m49.61372866s) [noise=true , address=77.90.31.17:4773 , piece_id=IGP6IK3ECIPSR4B4LM52HP5GNE2HLAULLHEZMOIASHDL3SQ7IQXA, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.742839164 (588h49m49.613922199s) [noise=true , address=77.90.31.14:4664 , piece_id=NUM5R3ZVTEBERIHLVJGPR3IMST2VZRSQZCBUEL7PZOV3V5Y4U3OA, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.742975143 (588h49m49.614058178s) [noise=true , address=145.79.227.41:4087 , piece_id=3FGUKPOY3737CHT3KT57FZS2WJB2WKP3P6A5LBZBBLBW5ND6MRMQ, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.743156177 (588h49m49.614239223s) [noise=true , address=145.79.227.23:4545 , piece_id=KHAZZ2XCMDL2B6IBIJBW2MMA7BZHF5GFK23TXK2RZTVBRMFHGGJA, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.743360346 (588h49m49.614443381s) [noise=true , address=145.79.227.51:4410 , piece_id=UBAFGGAMSLBFGZNKI6ZPYGJPP2INVXZLYSOFV4LVPW7M64MUNIXA, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.743467831 (588h49m49.614550876s) [noise=true , address=45.91.116.4:4939 , piece_id=OGUPZYEOVLRQP76YN2RA44LN6WYPAXQG3EESZCZEG4SL4CCWVCMQ, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.743791938 (588h49m49.614874973s) [noise=true , address=145.79.227.16:4909 , piece_id=7AZMPOQBFH5OQDJO4W5PGUZK7IV3ZMTJWHBN6L7AQKI7XNUU5A6A, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.744118821 (588h49m49.615201856s) [noise=true , address=45.91.116.9:4261 , piece_id=7CIUZUZ3WZWFY67YXUREKGQZK2C3VYCZCTICRAKUNSECYOFDBN2Q, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:51.437365305 (588h49m51.30844835s) [upload=516191]: Next returned nil segment Dec 30 04:18:51.437383689 (588h49m51.308466734s) [upload=516191]: Waiting for error group managing segments... Dec 30 04:18:51.728345062 (588h49m51.599428107s) [noise=true , address=145.79.227.12:4767 , piece_id=MNSOU6E3LNQDRP35J54JQJW5EX5ED4IHU5W3ANRIERMQ6K6YW44Q, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:51.728533030 (588h49m51.599616075s) [noise=true , address=145.79.227.11:4731 , piece_id=PG43LBI62W6NGHO5B64JM5HBC7DCHO73BQGIURAB2KJTCTX73F3Q, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:51.731150144 (588h49m51.602233179s) [noise=true , address=145.79.227.41:4087 , piece_id=3FGUKPOY3737CHT3KT57FZS2WJB2WKP3P6A5LBZBBLBW5ND6MRMQ, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:51.734456670 (588h49m51.605539725s) [noise=true , address=145.79.227.42:4122 , piece_id=6MSZTZ3U573XBLDLJGHCUH5TEO2ARLU72CZROAJ7MY3EX4HFJUJA, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:51.754644214 (588h49m51.625727259s) [noise=true , address=145.79.227.16:4909 , piece_id=7AZMPOQBFH5OQDJO4W5PGUZK7IV3ZMTJWHBN6L7AQKI7XNUU5A6A, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:51.764750749 (588h49m51.635833804s) [noise=true , address=145.79.227.23:4545 , piece_id=KHAZZ2XCMDL2B6IBIJBW2MMA7BZHF5GFK23TXK2RZTVBRMFHGGJA, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:51.765146684 (588h49m51.636229729s) [noise=true , address=145.79.227.51:4410 , piece_id=UBAFGGAMSLBFGZNKI6ZPYGJPP2INVXZLYSOFV4LVPW7M64MUNIXA, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:51.765488794 (588h49m51.636571829s) [noise=true , address=145.79.227.2:4115 , piece_id=JGVQEXNFDE7VLALIUASO5XYGG6Y7JGZ7ITJPLUUU6EBPDVV3BH5A, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:51.796376934 (588h49m51.667459969s) [noise=true , address=145.79.227.5:4223 , piece_id=LMEDBWW3UGMVXBBGQPJ5R66JRLMGYSJPMCCIBUZU5Y53SSXHJ3GA, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:52.032206050 (588h49m51.903289095s) [noise=true , address=45.91.116.4:4939 , piece_id=OGUPZYEOVLRQP76YN2RA44LN6WYPAXQG3EESZCZEG4SL4CCWVCMQ, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:52.132402783 (588h49m52.003485828s) [noise=true , address=45.91.116.9:4261 , piece_id=7CIUZUZ3WZWFY67YXUREKGQZK2C3VYCZCTICRAKUNSECYOFDBN2Q, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:52.132912924 (588h49m52.003995969s) [noise=true , address=77.90.31.14:4664 , piece_id=NUM5R3ZVTEBERIHLVJGPR3IMST2VZRSQZCBUEL7PZOV3V5Y4U3OA, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:52.232438690 (588h49m52.103521735s) [noise=true , address=77.90.31.18:4808 , piece_id=YUNEX6VTNVZJEQC573BCO253ZQRCTAA4Z2ZREVQOR26FKI2EQTTA, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:52.232460912 (588h49m52.103543947s) [seg_pos={0 0}, upload=516191]: Segment reached optimal threshold of 13 pieces. Dec 30 04:18:52.232659821 (588h49m52.103742866s) [noise=true , address=77.90.31.11:4553 , piece_id=UC7ZZLTIGMALZA5LFVRJ6F6CCS4UKLDHKE2HGR3ZVSG64W6U5K2A, seg_pos={0 0}, upload=516191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 30 04:18:52.232665572 (588h49m52.103748617s) [noise=true , address=77.90.31.11:4553 , piece_id=UC7ZZLTIGMALZA5LFVRJ6F6CCS4UKLDHKE2HGR3ZVSG64W6U5K2A, seg_pos={0 0}, upload=516191]: Long tail cancellation due to optimal threshold Dec 30 04:18:52.232942900 (588h49m52.104025945s) [noise=true , address=145.79.227.43:4157 , piece_id=6C7WJTZOUCQYIOURVSBMUIRIOTUY5TAMOTCWQ5CXNPEOV7CONDHA, seg_pos={0 0}, upload=516191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 04:18:52.232951837 (588h49m52.104034892s) [noise=true , address=145.79.227.43:4157 , piece_id=6C7WJTZOUCQYIOURVSBMUIRIOTUY5TAMOTCWQ5CXNPEOV7CONDHA, seg_pos={0 0}, upload=516191]: Long tail cancellation due to optimal threshold Dec 30 04:18:52.233247881 (588h49m52.104330916s) [noise=true , address=77.90.31.17:4773 , piece_id=IGP6IK3ECIPSR4B4LM52HP5GNE2HLAULLHEZMOIASHDL3SQ7IQXA, seg_pos={0 0}, upload=516191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: read tcp 188.166.225.170:43484->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:18:52.233256507 (588h49m52.104339552s) [noise=true , address=77.90.31.17:4773 , piece_id=IGP6IK3ECIPSR4B4LM52HP5GNE2HLAULLHEZMOIASHDL3SQ7IQXA, seg_pos={0 0}, upload=516191]: Long tail cancellation due to optimal threshold Dec 30 04:18:52.233279652 (588h49m52.104362707s) [seg_pos={0 0}, upload=516191]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:18:52.233372428 (588h49m52.104455473s) [upload=516191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:18:52.233380292 (588h49m52.104463348s) [upload=516191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:18:52.532231982 (588h49m52.403315017s) [upload=516191]: Done uploading segments... Dec 30 04:18:52.532269783 (588h49m52.403352818s) [upload=516191]: Upload finished. err: <nil> Duration: 5.922443168s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...XX.XX...XXXX...XXXX...X.X...XX..X.X..XX.......XX..X.X...X] duration: 30.002905114s trace: https://jaeger.dev.storj.io/trace/24d3b8beaf199c44 FAIL (2025-12-30 04:19:18 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 04:18:49.496068011 (588h49m49.367151056s) [upload=516191]: Starting upload Dec 30 04:18:49.496163933 (588h49m49.367246968s) [upload=516191]: Done starting upload Dec 30 04:18:49.496258503 (588h49m49.367341538s) [upload=516191]: Uploading segments... Dec 30 04:18:49.496534699 (588h49m49.367617744s) [upload=516191]: Got next segment. Inline: false Dec 30 04:18:49.496860750 (588h49m49.367943795s) [upload=516191]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:18:49.496867834 (588h49m49.367950869s) [upload=516191]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:18:49.739924243 (588h49m49.611007288s) [seg_pos={0 0}, upload=516191]: Begin upload segment... Dec 30 04:18:49.739981903 (588h49m49.611064948s) [seg_pos={0 0}, upload=516191]: Done begin upload segment. Dec 30 04:18:49.740032219 (588h49m49.611115264s) [noise=true , address=77.90.31.11:4553 , piece_id=UC7ZZLTIGMALZA5LFVRJ6F6CCS4UKLDHKE2HGR3ZVSG64W6U5K2A, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.740037448 (588h49m49.611120483s) [noise=true , address=145.79.227.11:4731 , piece_id=PG43LBI62W6NGHO5B64JM5HBC7DCHO73BQGIURAB2KJTCTX73F3Q, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.740784341 (588h49m49.611867386s) [noise=true , address=77.90.31.18:4808 , piece_id=YUNEX6VTNVZJEQC573BCO253ZQRCTAA4Z2ZREVQOR26FKI2EQTTA, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.741248845 (588h49m49.61233188s) [noise=true , address=145.79.227.12:4767 , piece_id=MNSOU6E3LNQDRP35J54JQJW5EX5ED4IHU5W3ANRIERMQ6K6YW44Q, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.741577111 (588h49m49.612660156s) [noise=true , address=145.79.227.2:4115 , piece_id=JGVQEXNFDE7VLALIUASO5XYGG6Y7JGZ7ITJPLUUU6EBPDVV3BH5A, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.741943278 (588h49m49.613026323s) [noise=true , address=145.79.227.5:4223 , piece_id=LMEDBWW3UGMVXBBGQPJ5R66JRLMGYSJPMCCIBUZU5Y53SSXHJ3GA, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.742230354 (588h49m49.613313399s) [noise=true , address=145.79.227.42:4122 , piece_id=6MSZTZ3U573XBLDLJGHCUH5TEO2ARLU72CZROAJ7MY3EX4HFJUJA, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.742502933 (588h49m49.613585978s) [noise=true , address=145.79.227.43:4157 , piece_id=6C7WJTZOUCQYIOURVSBMUIRIOTUY5TAMOTCWQ5CXNPEOV7CONDHA, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.742645615 (588h49m49.61372866s) [noise=true , address=77.90.31.17:4773 , piece_id=IGP6IK3ECIPSR4B4LM52HP5GNE2HLAULLHEZMOIASHDL3SQ7IQXA, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.742839164 (588h49m49.613922199s) [noise=true , address=77.90.31.14:4664 , piece_id=NUM5R3ZVTEBERIHLVJGPR3IMST2VZRSQZCBUEL7PZOV3V5Y4U3OA, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.742975143 (588h49m49.614058178s) [noise=true , address=145.79.227.41:4087 , piece_id=3FGUKPOY3737CHT3KT57FZS2WJB2WKP3P6A5LBZBBLBW5ND6MRMQ, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.743156177 (588h49m49.614239223s) [noise=true , address=145.79.227.23:4545 , piece_id=KHAZZ2XCMDL2B6IBIJBW2MMA7BZHF5GFK23TXK2RZTVBRMFHGGJA, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.743360346 (588h49m49.614443381s) [noise=true , address=145.79.227.51:4410 , piece_id=UBAFGGAMSLBFGZNKI6ZPYGJPP2INVXZLYSOFV4LVPW7M64MUNIXA, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.743467831 (588h49m49.614550876s) [noise=true , address=45.91.116.4:4939 , piece_id=OGUPZYEOVLRQP76YN2RA44LN6WYPAXQG3EESZCZEG4SL4CCWVCMQ, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.743791938 (588h49m49.614874973s) [noise=true , address=145.79.227.16:4909 , piece_id=7AZMPOQBFH5OQDJO4W5PGUZK7IV3ZMTJWHBN6L7AQKI7XNUU5A6A, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:49.744118821 (588h49m49.615201856s) [noise=true , address=45.91.116.9:4261 , piece_id=7CIUZUZ3WZWFY67YXUREKGQZK2C3VYCZCTICRAKUNSECYOFDBN2Q, seg_pos={0 0}, upload=516191]: Uploading piece... Dec 30 04:18:51.437365305 (588h49m51.30844835s) [upload=516191]: Next returned nil segment Dec 30 04:18:51.437383689 (588h49m51.308466734s) [upload=516191]: Waiting for error group managing segments... Dec 30 04:18:51.728345062 (588h49m51.599428107s) [noise=true , address=145.79.227.12:4767 , piece_id=MNSOU6E3LNQDRP35J54JQJW5EX5ED4IHU5W3ANRIERMQ6K6YW44Q, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:51.728533030 (588h49m51.599616075s) [noise=true , address=145.79.227.11:4731 , piece_id=PG43LBI62W6NGHO5B64JM5HBC7DCHO73BQGIURAB2KJTCTX73F3Q, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:51.731150144 (588h49m51.602233179s) [noise=true , address=145.79.227.41:4087 , piece_id=3FGUKPOY3737CHT3KT57FZS2WJB2WKP3P6A5LBZBBLBW5ND6MRMQ, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:51.734456670 (588h49m51.605539725s) [noise=true , address=145.79.227.42:4122 , piece_id=6MSZTZ3U573XBLDLJGHCUH5TEO2ARLU72CZROAJ7MY3EX4HFJUJA, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:51.754644214 (588h49m51.625727259s) [noise=true , address=145.79.227.16:4909 , piece_id=7AZMPOQBFH5OQDJO4W5PGUZK7IV3ZMTJWHBN6L7AQKI7XNUU5A6A, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:51.764750749 (588h49m51.635833804s) [noise=true , address=145.79.227.23:4545 , piece_id=KHAZZ2XCMDL2B6IBIJBW2MMA7BZHF5GFK23TXK2RZTVBRMFHGGJA, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:51.765146684 (588h49m51.636229729s) [noise=true , address=145.79.227.51:4410 , piece_id=UBAFGGAMSLBFGZNKI6ZPYGJPP2INVXZLYSOFV4LVPW7M64MUNIXA, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:51.765488794 (588h49m51.636571829s) [noise=true , address=145.79.227.2:4115 , piece_id=JGVQEXNFDE7VLALIUASO5XYGG6Y7JGZ7ITJPLUUU6EBPDVV3BH5A, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:51.796376934 (588h49m51.667459969s) [noise=true , address=145.79.227.5:4223 , piece_id=LMEDBWW3UGMVXBBGQPJ5R66JRLMGYSJPMCCIBUZU5Y53SSXHJ3GA, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:52.032206050 (588h49m51.903289095s) [noise=true , address=45.91.116.4:4939 , piece_id=OGUPZYEOVLRQP76YN2RA44LN6WYPAXQG3EESZCZEG4SL4CCWVCMQ, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:52.132402783 (588h49m52.003485828s) [noise=true , address=45.91.116.9:4261 , piece_id=7CIUZUZ3WZWFY67YXUREKGQZK2C3VYCZCTICRAKUNSECYOFDBN2Q, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:52.132912924 (588h49m52.003995969s) [noise=true , address=77.90.31.14:4664 , piece_id=NUM5R3ZVTEBERIHLVJGPR3IMST2VZRSQZCBUEL7PZOV3V5Y4U3OA, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:52.232438690 (588h49m52.103521735s) [noise=true , address=77.90.31.18:4808 , piece_id=YUNEX6VTNVZJEQC573BCO253ZQRCTAA4Z2ZREVQOR26FKI2EQTTA, seg_pos={0 0}, upload=516191]: Done uploading piece. err: <nil> Dec 30 04:18:52.232460912 (588h49m52.103543947s) [seg_pos={0 0}, upload=516191]: Segment reached optimal threshold of 13 pieces. Dec 30 04:18:52.232659821 (588h49m52.103742866s) [noise=true , address=77.90.31.11:4553 , piece_id=UC7ZZLTIGMALZA5LFVRJ6F6CCS4UKLDHKE2HGR3ZVSG64W6U5K2A, seg_pos={0 0}, upload=516191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF Dec 30 04:18:52.232665572 (588h49m52.103748617s) [noise=true , address=77.90.31.11:4553 , piece_id=UC7ZZLTIGMALZA5LFVRJ6F6CCS4UKLDHKE2HGR3ZVSG64W6U5K2A, seg_pos={0 0}, upload=516191]: Long tail cancellation due to optimal threshold Dec 30 04:18:52.232942900 (588h49m52.104025945s) [noise=true , address=145.79.227.43:4157 , piece_id=6C7WJTZOUCQYIOURVSBMUIRIOTUY5TAMOTCWQ5CXNPEOV7CONDHA, seg_pos={0 0}, upload=516191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz): protocol: expected piece hash; context canceled; EOF Dec 30 04:18:52.232951837 (588h49m52.104034892s) [noise=true , address=145.79.227.43:4157 , piece_id=6C7WJTZOUCQYIOURVSBMUIRIOTUY5TAMOTCWQ5CXNPEOV7CONDHA, seg_pos={0 0}, upload=516191]: Long tail cancellation due to optimal threshold Dec 30 04:18:52.233247881 (588h49m52.104330916s) [noise=true , address=77.90.31.17:4773 , piece_id=IGP6IK3ECIPSR4B4LM52HP5GNE2HLAULLHEZMOIASHDL3SQ7IQXA, seg_pos={0 0}, upload=516191]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: read tcp 188.166.225.170:43484->77.90.31.17:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:18:52.233256507 (588h49m52.104339552s) [noise=true , address=77.90.31.17:4773 , piece_id=IGP6IK3ECIPSR4B4LM52HP5GNE2HLAULLHEZMOIASHDL3SQ7IQXA, seg_pos={0 0}, upload=516191]: Long tail cancellation due to optimal threshold Dec 30 04:18:52.233279652 (588h49m52.104362707s) [seg_pos={0 0}, upload=516191]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:18:52.233372428 (588h49m52.104455473s) [upload=516191]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:18:52.233380292 (588h49m52.104463348s) [upload=516191]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:18:52.532231982 (588h49m52.403315017s) [upload=516191]: Done uploading segments... Dec 30 04:18:52.532269783 (588h49m52.403352818s) [upload=516191]: Upload finished. err: <nil> Duration: 30.002927387s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.....................X..XXX............XX..X.....X..........] duration: 3.524484732s trace: https://jaeger.dev.storj.io/trace/408434392b868da5 FAIL (2025-12-30 04:19:05 UTC) orbiter warning: context deadline exceeded Duration: 3.524572029s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [......................X..XXX............XX..X.....X.........] duration: 4.921683579s trace: https://jaeger.dev.storj.io/trace/408434392b868da5 FAIL (2025-12-30 04:18:44 UTC) orbiter warning: context deadline exceeded Duration: 4.921725209s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X............X.....................] duration: 37.653078045s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:18:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 37.653112541s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [....XX.XX...XXXX...XXXX...X.X...XX..X.X..XX.......XX..X.X...] duration: 6.72179552s trace: https://jaeger.dev.storj.io/trace/52a8d1ed6334dda7 FAIL (2025-12-30 04:18:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 2I5I3A3DHEUL532UZROI5NPTOLL5WTA7VTMXSPOE5DFJFUCFVU2A) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: PSUNGGNF7EXLTZFXNUEWINIWFOS4X2IXP67RSKCIUYYUHU5YAH5Q) context deadline exceeded ======LOGS====== 88h46m21.618666665s) [upload=516142]: Done starting upload Dec 30 04:15:21.747683661 (588h46m21.618766706s) [upload=516142]: Uploading segments... Dec 30 04:15:21.747801685 (588h46m21.61888472s) [upload=516142]: Got next segment. Inline: false Dec 30 04:15:21.747835690 (588h46m21.618918725s) [upload=516142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:15:21.747840569 (588h46m21.618923604s) [upload=516142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:15:22.002365677 (588h46m21.873448742s) [seg_pos={0 0}, upload=516142]: Begin upload segment... Dec 30 04:15:22.002426603 (588h46m21.873509648s) [seg_pos={0 0}, upload=516142]: Done begin upload segment. Dec 30 04:15:22.002498871 (588h46m21.873581906s) [noise=true , address=77.90.31.11:4555 , piece_id=ZSTA7TWF2SQYUYSB5GACGEENZVOFTN27GQX7IUUDKUYHIZN6K2XA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.002514872 (588h46m21.873597917s) [noise=true , address=145.79.227.5:4222 , piece_id=YMRSUP4MY2Q2W6MCISFDS3HQUWQN6HXEGMVZ54F5WU27HKQUACIQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.003130323 (588h46m21.874213368s) [noise=true , address=145.79.227.16:4909 , piece_id=QGRJWA2HW7SSJFY525DADV5TIDHAR3KFDMO3SNC3OAVBIYKGWUQQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.003452718 (588h46m21.874535763s) [noise=true , address=145.79.227.4:4187 , piece_id=ISJVYSCF5VDDKDROWFAS32EK7UTNOZKR7ULTXNKUHR2GS2JDI5XQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.003877076 (588h46m21.874960121s) [noise=true , address=45.91.116.10:4081 , piece_id=SFMDWU7SYBSVWQXZHFD7GNLZPX4SRRARFBJ22WJA3D4MDGBMJSRA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.004385774 (588h46m21.875468819s) [noise=true , address=145.79.227.3:4150 , piece_id=FPJ362RBQ4GAROMSUMPJ3TKFNUBEPX5XYYA6ATM6XGI6BNAUH3YQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.004840390 (588h46m21.875923425s) [noise=true , address=45.91.116.11:4119 , piece_id=5L25DQXXPYCMYIKBH5NRVA4V567NAUMMSM67FZ3I7A5YGA6V4G3Q, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.005316576 (588h46m21.876399611s) [noise=true , address=145.79.227.37:4566 , piece_id=OGJWRCWWMNUUYKOTOJTN2MDDVPEGO633J2H73CWATPGU5QJ3SKGA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.005970341 (588h46m21.877053386s) [noise=true , address=145.79.227.46:4601 , piece_id=3YNOCOUWS665FJD27U34OCSZQY7HMSZEXX3YPSWVM5S6T26NZ5FQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.006259603 (588h46m21.877342668s) [noise=true , address=145.79.227.15:4874 , piece_id=SCU6DJ7H5GA2HTE6XMMBOCA4JTMKDKEAHVHI4AP5UZMANMNDDUXQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.006558952 (588h46m21.877641987s) [noise=true , address=45.91.116.6:4975 , piece_id=2I5I3A3DHEUL532UZROI5NPTOLL5WTA7VTMXSPOE5DFJFUCFVU2A, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.007047553 (588h46m21.878130588s) [noise=true , address=77.90.31.21:4878 , piece_id=NVWUX54T3OSYMCJIF62LFX5ZUHWKXF76OL5XCWBZKWIIXYQGAH4Q, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.007620815 (588h46m21.87870386s) [noise=true , address=145.79.227.47:4265 , piece_id=7JTB2NTMH3CAT7ESWHFR2JRSXZGYKZI6CVHZFO3B44GJ5UIK6UMA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.008172646 (588h46m21.879255681s) [noise=true , address=145.79.227.20:4439 , piece_id=FBDGCBQ6NAURG4BNXWOVQNUABWLTBOFVA6UXF556LSEJF3JFLAYA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.008749674 (588h46m21.879832749s) [noise=true , address=145.79.227.11:4729 , piece_id=OXWEVOXVQBJZNTIXOPJYGPPM4XYU7T5VW6PA4K7BCCTB4VDQV3KQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.008882637 (588h46m21.879965682s) [noise=true , address=77.90.31.16:4736 , piece_id=PSUNGGNF7EXLTZFXNUEWINIWFOS4X2IXP67RSKCIUYYUHU5YAH5Q, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:23.711185336 (588h46m23.582268381s) [upload=516142]: Next returned nil segment Dec 30 04:15:23.711219642 (588h46m23.582302677s) [upload=516142]: Waiting for error group managing segments... Dec 30 04:15:23.993757614 (588h46m23.864840659s) [noise=true , address=145.79.227.3:4150 , piece_id=FPJ362RBQ4GAROMSUMPJ3TKFNUBEPX5XYYA6ATM6XGI6BNAUH3YQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:23.995291104 (588h46m23.866374149s) [noise=true , address=145.79.227.15:4874 , piece_id=SCU6DJ7H5GA2HTE6XMMBOCA4JTMKDKEAHVHI4AP5UZMANMNDDUXQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:23.997226881 (588h46m23.868309926s) [noise=true , address=145.79.227.16:4909 , piece_id=QGRJWA2HW7SSJFY525DADV5TIDHAR3KFDMO3SNC3OAVBIYKGWUQQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:23.997333694 (588h46m23.868416739s) [noise=true , address=145.79.227.20:4439 , piece_id=FBDGCBQ6NAURG4BNXWOVQNUABWLTBOFVA6UXF556LSEJF3JFLAYA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:23.999259902 (588h46m23.870342947s) [noise=true , address=145.79.227.11:4729 , piece_id=OXWEVOXVQBJZNTIXOPJYGPPM4XYU7T5VW6PA4K7BCCTB4VDQV3KQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.000377961 (588h46m23.871461007s) [noise=true , address=145.79.227.5:4222 , piece_id=YMRSUP4MY2Q2W6MCISFDS3HQUWQN6HXEGMVZ54F5WU27HKQUACIQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.010766804 (588h46m23.881849839s) [noise=true , address=145.79.227.46:4601 , piece_id=3YNOCOUWS665FJD27U34OCSZQY7HMSZEXX3YPSWVM5S6T26NZ5FQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.016615803 (588h46m23.887698848s) [noise=true , address=145.79.227.47:4265 , piece_id=7JTB2NTMH3CAT7ESWHFR2JRSXZGYKZI6CVHZFO3B44GJ5UIK6UMA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.021063222 (588h46m23.892146257s) [noise=true , address=145.79.227.4:4187 , piece_id=ISJVYSCF5VDDKDROWFAS32EK7UTNOZKR7ULTXNKUHR2GS2JDI5XQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.035011132 (588h46m23.906094177s) [noise=true , address=145.79.227.37:4566 , piece_id=OGJWRCWWMNUUYKOTOJTN2MDDVPEGO633J2H73CWATPGU5QJ3SKGA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.273450619 (588h46m24.144533654s) [noise=true , address=77.90.31.16:4736 , piece_id=PSUNGGNF7EXLTZFXNUEWINIWFOS4X2IXP67RSKCIUYYUHU5YAH5Q, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.309222650 (588h46m24.180305685s) [noise=true , address=77.90.31.11:4555 , piece_id=ZSTA7TWF2SQYUYSB5GACGEENZVOFTN27GQX7IUUDKUYHIZN6K2XA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.404781931 (588h46m24.275864976s) [noise=true , address=45.91.116.6:4975 , piece_id=2I5I3A3DHEUL532UZROI5NPTOLL5WTA7VTMXSPOE5DFJFUCFVU2A, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.404812499 (588h46m24.275895535s) [seg_pos={0 0}, upload=516142]: Segment reached optimal threshold of 13 pieces. Dec 30 04:15:24.405079347 (588h46m24.276162422s) [noise=true , address=77.90.31.21:4878 , piece_id=NVWUX54T3OSYMCJIF62LFX5ZUHWKXF76OL5XCWBZKWIIXYQGAH4Q, seg_pos={0 0}, upload=516142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:47932->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:47932->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47932->77.90.31.21:4878: use of closed network connection Dec 30 04:15:24.405088976 (588h46m24.276172021s) [noise=true , address=77.90.31.21:4878 , piece_id=NVWUX54T3OSYMCJIF62LFX5ZUHWKXF76OL5XCWBZKWIIXYQGAH4Q, seg_pos={0 0}, upload=516142]: Long tail cancellation due to optimal threshold Dec 30 04:15:24.405393706 (588h46m24.276476742s) [noise=true , address=45.91.116.11:4119 , piece_id=5L25DQXXPYCMYIKBH5NRVA4V567NAUMMSM67FZ3I7A5YGA6V4G3Q, seg_pos={0 0}, upload=516142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 30 04:15:24.405400740 (588h46m24.276483775s) [noise=true , address=45.91.116.11:4119 , piece_id=5L25DQXXPYCMYIKBH5NRVA4V567NAUMMSM67FZ3I7A5YGA6V4G3Q, seg_pos={0 0}, upload=516142]: Long tail cancellation due to optimal threshold Dec 30 04:15:24.405611200 (588h46m24.276694235s) [noise=true , address=45.91.116.10:4081 , piece_id=SFMDWU7SYBSVWQXZHFD7GNLZPX4SRRARFBJ22WJA3D4MDGBMJSRA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: read tcp 188.166.225.170:50536->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:15:24.405633212 (588h46m24.276716267s) [noise=true , address=45.91.116.10:4081 , piece_id=SFMDWU7SYBSVWQXZHFD7GNLZPX4SRRARFBJ22WJA3D4MDGBMJSRA, seg_pos={0 0}, upload=516142]: Long tail cancellation due to optimal threshold Dec 30 04:15:24.405666535 (588h46m24.27674958s) [seg_pos={0 0}, upload=516142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:15:24.405857599 (588h46m24.276940645s) [upload=516142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:15:24.405867458 (588h46m24.276950503s) [upload=516142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:15:24.703060479 (588h46m24.574143514s) [upload=516142]: Done uploading segments... Dec 30 04:15:24.703086849 (588h46m24.574169884s) [upload=516142]: Upload finished. err: <nil> Duration: 6.721840085s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.......................X..XXX............XX..X.....X........] duration: 3.570427891s trace: https://jaeger.dev.storj.io/trace/408434392b868da5 FAIL (2025-12-30 04:17:49 UTC) orbiter warning: context deadline exceeded Duration: 3.570460204s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [X.......................X..XXX............XX..X.....X.......] duration: 3.480550781s trace: https://jaeger.dev.storj.io/trace/408434392b868da5 FAIL (2025-12-30 04:17:11 UTC) orbiter warning: context deadline exceeded Duration: 3.480590818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.....XX.XX...XXXX...XXXX...X.X...XX..X.X..XX.......XX..X.X..] duration: 5.763715156s trace: https://jaeger.dev.storj.io/trace/52a8d1ed6334dda7 FAIL (2025-12-30 04:17:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 2I5I3A3DHEUL532UZROI5NPTOLL5WTA7VTMXSPOE5DFJFUCFVU2A) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: PSUNGGNF7EXLTZFXNUEWINIWFOS4X2IXP67RSKCIUYYUHU5YAH5Q) context deadline exceeded ======LOGS====== 88h46m21.618666665s) [upload=516142]: Done starting upload Dec 30 04:15:21.747683661 (588h46m21.618766706s) [upload=516142]: Uploading segments... Dec 30 04:15:21.747801685 (588h46m21.61888472s) [upload=516142]: Got next segment. Inline: false Dec 30 04:15:21.747835690 (588h46m21.618918725s) [upload=516142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:15:21.747840569 (588h46m21.618923604s) [upload=516142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:15:22.002365677 (588h46m21.873448742s) [seg_pos={0 0}, upload=516142]: Begin upload segment... Dec 30 04:15:22.002426603 (588h46m21.873509648s) [seg_pos={0 0}, upload=516142]: Done begin upload segment. Dec 30 04:15:22.002498871 (588h46m21.873581906s) [noise=true , address=77.90.31.11:4555 , piece_id=ZSTA7TWF2SQYUYSB5GACGEENZVOFTN27GQX7IUUDKUYHIZN6K2XA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.002514872 (588h46m21.873597917s) [noise=true , address=145.79.227.5:4222 , piece_id=YMRSUP4MY2Q2W6MCISFDS3HQUWQN6HXEGMVZ54F5WU27HKQUACIQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.003130323 (588h46m21.874213368s) [noise=true , address=145.79.227.16:4909 , piece_id=QGRJWA2HW7SSJFY525DADV5TIDHAR3KFDMO3SNC3OAVBIYKGWUQQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.003452718 (588h46m21.874535763s) [noise=true , address=145.79.227.4:4187 , piece_id=ISJVYSCF5VDDKDROWFAS32EK7UTNOZKR7ULTXNKUHR2GS2JDI5XQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.003877076 (588h46m21.874960121s) [noise=true , address=45.91.116.10:4081 , piece_id=SFMDWU7SYBSVWQXZHFD7GNLZPX4SRRARFBJ22WJA3D4MDGBMJSRA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.004385774 (588h46m21.875468819s) [noise=true , address=145.79.227.3:4150 , piece_id=FPJ362RBQ4GAROMSUMPJ3TKFNUBEPX5XYYA6ATM6XGI6BNAUH3YQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.004840390 (588h46m21.875923425s) [noise=true , address=45.91.116.11:4119 , piece_id=5L25DQXXPYCMYIKBH5NRVA4V567NAUMMSM67FZ3I7A5YGA6V4G3Q, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.005316576 (588h46m21.876399611s) [noise=true , address=145.79.227.37:4566 , piece_id=OGJWRCWWMNUUYKOTOJTN2MDDVPEGO633J2H73CWATPGU5QJ3SKGA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.005970341 (588h46m21.877053386s) [noise=true , address=145.79.227.46:4601 , piece_id=3YNOCOUWS665FJD27U34OCSZQY7HMSZEXX3YPSWVM5S6T26NZ5FQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.006259603 (588h46m21.877342668s) [noise=true , address=145.79.227.15:4874 , piece_id=SCU6DJ7H5GA2HTE6XMMBOCA4JTMKDKEAHVHI4AP5UZMANMNDDUXQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.006558952 (588h46m21.877641987s) [noise=true , address=45.91.116.6:4975 , piece_id=2I5I3A3DHEUL532UZROI5NPTOLL5WTA7VTMXSPOE5DFJFUCFVU2A, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.007047553 (588h46m21.878130588s) [noise=true , address=77.90.31.21:4878 , piece_id=NVWUX54T3OSYMCJIF62LFX5ZUHWKXF76OL5XCWBZKWIIXYQGAH4Q, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.007620815 (588h46m21.87870386s) [noise=true , address=145.79.227.47:4265 , piece_id=7JTB2NTMH3CAT7ESWHFR2JRSXZGYKZI6CVHZFO3B44GJ5UIK6UMA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.008172646 (588h46m21.879255681s) [noise=true , address=145.79.227.20:4439 , piece_id=FBDGCBQ6NAURG4BNXWOVQNUABWLTBOFVA6UXF556LSEJF3JFLAYA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.008749674 (588h46m21.879832749s) [noise=true , address=145.79.227.11:4729 , piece_id=OXWEVOXVQBJZNTIXOPJYGPPM4XYU7T5VW6PA4K7BCCTB4VDQV3KQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.008882637 (588h46m21.879965682s) [noise=true , address=77.90.31.16:4736 , piece_id=PSUNGGNF7EXLTZFXNUEWINIWFOS4X2IXP67RSKCIUYYUHU5YAH5Q, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:23.711185336 (588h46m23.582268381s) [upload=516142]: Next returned nil segment Dec 30 04:15:23.711219642 (588h46m23.582302677s) [upload=516142]: Waiting for error group managing segments... Dec 30 04:15:23.993757614 (588h46m23.864840659s) [noise=true , address=145.79.227.3:4150 , piece_id=FPJ362RBQ4GAROMSUMPJ3TKFNUBEPX5XYYA6ATM6XGI6BNAUH3YQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:23.995291104 (588h46m23.866374149s) [noise=true , address=145.79.227.15:4874 , piece_id=SCU6DJ7H5GA2HTE6XMMBOCA4JTMKDKEAHVHI4AP5UZMANMNDDUXQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:23.997226881 (588h46m23.868309926s) [noise=true , address=145.79.227.16:4909 , piece_id=QGRJWA2HW7SSJFY525DADV5TIDHAR3KFDMO3SNC3OAVBIYKGWUQQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:23.997333694 (588h46m23.868416739s) [noise=true , address=145.79.227.20:4439 , piece_id=FBDGCBQ6NAURG4BNXWOVQNUABWLTBOFVA6UXF556LSEJF3JFLAYA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:23.999259902 (588h46m23.870342947s) [noise=true , address=145.79.227.11:4729 , piece_id=OXWEVOXVQBJZNTIXOPJYGPPM4XYU7T5VW6PA4K7BCCTB4VDQV3KQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.000377961 (588h46m23.871461007s) [noise=true , address=145.79.227.5:4222 , piece_id=YMRSUP4MY2Q2W6MCISFDS3HQUWQN6HXEGMVZ54F5WU27HKQUACIQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.010766804 (588h46m23.881849839s) [noise=true , address=145.79.227.46:4601 , piece_id=3YNOCOUWS665FJD27U34OCSZQY7HMSZEXX3YPSWVM5S6T26NZ5FQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.016615803 (588h46m23.887698848s) [noise=true , address=145.79.227.47:4265 , piece_id=7JTB2NTMH3CAT7ESWHFR2JRSXZGYKZI6CVHZFO3B44GJ5UIK6UMA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.021063222 (588h46m23.892146257s) [noise=true , address=145.79.227.4:4187 , piece_id=ISJVYSCF5VDDKDROWFAS32EK7UTNOZKR7ULTXNKUHR2GS2JDI5XQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.035011132 (588h46m23.906094177s) [noise=true , address=145.79.227.37:4566 , piece_id=OGJWRCWWMNUUYKOTOJTN2MDDVPEGO633J2H73CWATPGU5QJ3SKGA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.273450619 (588h46m24.144533654s) [noise=true , address=77.90.31.16:4736 , piece_id=PSUNGGNF7EXLTZFXNUEWINIWFOS4X2IXP67RSKCIUYYUHU5YAH5Q, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.309222650 (588h46m24.180305685s) [noise=true , address=77.90.31.11:4555 , piece_id=ZSTA7TWF2SQYUYSB5GACGEENZVOFTN27GQX7IUUDKUYHIZN6K2XA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.404781931 (588h46m24.275864976s) [noise=true , address=45.91.116.6:4975 , piece_id=2I5I3A3DHEUL532UZROI5NPTOLL5WTA7VTMXSPOE5DFJFUCFVU2A, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.404812499 (588h46m24.275895535s) [seg_pos={0 0}, upload=516142]: Segment reached optimal threshold of 13 pieces. Dec 30 04:15:24.405079347 (588h46m24.276162422s) [noise=true , address=77.90.31.21:4878 , piece_id=NVWUX54T3OSYMCJIF62LFX5ZUHWKXF76OL5XCWBZKWIIXYQGAH4Q, seg_pos={0 0}, upload=516142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:47932->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:47932->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47932->77.90.31.21:4878: use of closed network connection Dec 30 04:15:24.405088976 (588h46m24.276172021s) [noise=true , address=77.90.31.21:4878 , piece_id=NVWUX54T3OSYMCJIF62LFX5ZUHWKXF76OL5XCWBZKWIIXYQGAH4Q, seg_pos={0 0}, upload=516142]: Long tail cancellation due to optimal threshold Dec 30 04:15:24.405393706 (588h46m24.276476742s) [noise=true , address=45.91.116.11:4119 , piece_id=5L25DQXXPYCMYIKBH5NRVA4V567NAUMMSM67FZ3I7A5YGA6V4G3Q, seg_pos={0 0}, upload=516142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 30 04:15:24.405400740 (588h46m24.276483775s) [noise=true , address=45.91.116.11:4119 , piece_id=5L25DQXXPYCMYIKBH5NRVA4V567NAUMMSM67FZ3I7A5YGA6V4G3Q, seg_pos={0 0}, upload=516142]: Long tail cancellation due to optimal threshold Dec 30 04:15:24.405611200 (588h46m24.276694235s) [noise=true , address=45.91.116.10:4081 , piece_id=SFMDWU7SYBSVWQXZHFD7GNLZPX4SRRARFBJ22WJA3D4MDGBMJSRA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: read tcp 188.166.225.170:50536->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:15:24.405633212 (588h46m24.276716267s) [noise=true , address=45.91.116.10:4081 , piece_id=SFMDWU7SYBSVWQXZHFD7GNLZPX4SRRARFBJ22WJA3D4MDGBMJSRA, seg_pos={0 0}, upload=516142]: Long tail cancellation due to optimal threshold Dec 30 04:15:24.405666535 (588h46m24.27674958s) [seg_pos={0 0}, upload=516142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:15:24.405857599 (588h46m24.276940645s) [upload=516142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:15:24.405867458 (588h46m24.276950503s) [upload=516142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:15:24.703060479 (588h46m24.574143514s) [upload=516142]: Done uploading segments... Dec 30 04:15:24.703086849 (588h46m24.574169884s) [upload=516142]: Upload finished. err: <nil> Duration: 5.763740755s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X............X....................] duration: 15.301468832s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:16:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 15.301490676s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X............X...................] duration: 44.880080365s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:16:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 44.880098341s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [......XX.XX...XXXX...XXXX...X.X...XX..X.X..XX.......XX..X.X.] duration: 5.866016906s trace: https://jaeger.dev.storj.io/trace/52a8d1ed6334dda7 FAIL (2025-12-30 04:16:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 2I5I3A3DHEUL532UZROI5NPTOLL5WTA7VTMXSPOE5DFJFUCFVU2A) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: PSUNGGNF7EXLTZFXNUEWINIWFOS4X2IXP67RSKCIUYYUHU5YAH5Q) context deadline exceeded ======LOGS====== 88h46m21.618666665s) [upload=516142]: Done starting upload Dec 30 04:15:21.747683661 (588h46m21.618766706s) [upload=516142]: Uploading segments... Dec 30 04:15:21.747801685 (588h46m21.61888472s) [upload=516142]: Got next segment. Inline: false Dec 30 04:15:21.747835690 (588h46m21.618918725s) [upload=516142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:15:21.747840569 (588h46m21.618923604s) [upload=516142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:15:22.002365677 (588h46m21.873448742s) [seg_pos={0 0}, upload=516142]: Begin upload segment... Dec 30 04:15:22.002426603 (588h46m21.873509648s) [seg_pos={0 0}, upload=516142]: Done begin upload segment. Dec 30 04:15:22.002498871 (588h46m21.873581906s) [noise=true , address=77.90.31.11:4555 , piece_id=ZSTA7TWF2SQYUYSB5GACGEENZVOFTN27GQX7IUUDKUYHIZN6K2XA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.002514872 (588h46m21.873597917s) [noise=true , address=145.79.227.5:4222 , piece_id=YMRSUP4MY2Q2W6MCISFDS3HQUWQN6HXEGMVZ54F5WU27HKQUACIQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.003130323 (588h46m21.874213368s) [noise=true , address=145.79.227.16:4909 , piece_id=QGRJWA2HW7SSJFY525DADV5TIDHAR3KFDMO3SNC3OAVBIYKGWUQQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.003452718 (588h46m21.874535763s) [noise=true , address=145.79.227.4:4187 , piece_id=ISJVYSCF5VDDKDROWFAS32EK7UTNOZKR7ULTXNKUHR2GS2JDI5XQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.003877076 (588h46m21.874960121s) [noise=true , address=45.91.116.10:4081 , piece_id=SFMDWU7SYBSVWQXZHFD7GNLZPX4SRRARFBJ22WJA3D4MDGBMJSRA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.004385774 (588h46m21.875468819s) [noise=true , address=145.79.227.3:4150 , piece_id=FPJ362RBQ4GAROMSUMPJ3TKFNUBEPX5XYYA6ATM6XGI6BNAUH3YQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.004840390 (588h46m21.875923425s) [noise=true , address=45.91.116.11:4119 , piece_id=5L25DQXXPYCMYIKBH5NRVA4V567NAUMMSM67FZ3I7A5YGA6V4G3Q, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.005316576 (588h46m21.876399611s) [noise=true , address=145.79.227.37:4566 , piece_id=OGJWRCWWMNUUYKOTOJTN2MDDVPEGO633J2H73CWATPGU5QJ3SKGA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.005970341 (588h46m21.877053386s) [noise=true , address=145.79.227.46:4601 , piece_id=3YNOCOUWS665FJD27U34OCSZQY7HMSZEXX3YPSWVM5S6T26NZ5FQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.006259603 (588h46m21.877342668s) [noise=true , address=145.79.227.15:4874 , piece_id=SCU6DJ7H5GA2HTE6XMMBOCA4JTMKDKEAHVHI4AP5UZMANMNDDUXQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.006558952 (588h46m21.877641987s) [noise=true , address=45.91.116.6:4975 , piece_id=2I5I3A3DHEUL532UZROI5NPTOLL5WTA7VTMXSPOE5DFJFUCFVU2A, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.007047553 (588h46m21.878130588s) [noise=true , address=77.90.31.21:4878 , piece_id=NVWUX54T3OSYMCJIF62LFX5ZUHWKXF76OL5XCWBZKWIIXYQGAH4Q, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.007620815 (588h46m21.87870386s) [noise=true , address=145.79.227.47:4265 , piece_id=7JTB2NTMH3CAT7ESWHFR2JRSXZGYKZI6CVHZFO3B44GJ5UIK6UMA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.008172646 (588h46m21.879255681s) [noise=true , address=145.79.227.20:4439 , piece_id=FBDGCBQ6NAURG4BNXWOVQNUABWLTBOFVA6UXF556LSEJF3JFLAYA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.008749674 (588h46m21.879832749s) [noise=true , address=145.79.227.11:4729 , piece_id=OXWEVOXVQBJZNTIXOPJYGPPM4XYU7T5VW6PA4K7BCCTB4VDQV3KQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.008882637 (588h46m21.879965682s) [noise=true , address=77.90.31.16:4736 , piece_id=PSUNGGNF7EXLTZFXNUEWINIWFOS4X2IXP67RSKCIUYYUHU5YAH5Q, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:23.711185336 (588h46m23.582268381s) [upload=516142]: Next returned nil segment Dec 30 04:15:23.711219642 (588h46m23.582302677s) [upload=516142]: Waiting for error group managing segments... Dec 30 04:15:23.993757614 (588h46m23.864840659s) [noise=true , address=145.79.227.3:4150 , piece_id=FPJ362RBQ4GAROMSUMPJ3TKFNUBEPX5XYYA6ATM6XGI6BNAUH3YQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:23.995291104 (588h46m23.866374149s) [noise=true , address=145.79.227.15:4874 , piece_id=SCU6DJ7H5GA2HTE6XMMBOCA4JTMKDKEAHVHI4AP5UZMANMNDDUXQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:23.997226881 (588h46m23.868309926s) [noise=true , address=145.79.227.16:4909 , piece_id=QGRJWA2HW7SSJFY525DADV5TIDHAR3KFDMO3SNC3OAVBIYKGWUQQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:23.997333694 (588h46m23.868416739s) [noise=true , address=145.79.227.20:4439 , piece_id=FBDGCBQ6NAURG4BNXWOVQNUABWLTBOFVA6UXF556LSEJF3JFLAYA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:23.999259902 (588h46m23.870342947s) [noise=true , address=145.79.227.11:4729 , piece_id=OXWEVOXVQBJZNTIXOPJYGPPM4XYU7T5VW6PA4K7BCCTB4VDQV3KQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.000377961 (588h46m23.871461007s) [noise=true , address=145.79.227.5:4222 , piece_id=YMRSUP4MY2Q2W6MCISFDS3HQUWQN6HXEGMVZ54F5WU27HKQUACIQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.010766804 (588h46m23.881849839s) [noise=true , address=145.79.227.46:4601 , piece_id=3YNOCOUWS665FJD27U34OCSZQY7HMSZEXX3YPSWVM5S6T26NZ5FQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.016615803 (588h46m23.887698848s) [noise=true , address=145.79.227.47:4265 , piece_id=7JTB2NTMH3CAT7ESWHFR2JRSXZGYKZI6CVHZFO3B44GJ5UIK6UMA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.021063222 (588h46m23.892146257s) [noise=true , address=145.79.227.4:4187 , piece_id=ISJVYSCF5VDDKDROWFAS32EK7UTNOZKR7ULTXNKUHR2GS2JDI5XQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.035011132 (588h46m23.906094177s) [noise=true , address=145.79.227.37:4566 , piece_id=OGJWRCWWMNUUYKOTOJTN2MDDVPEGO633J2H73CWATPGU5QJ3SKGA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.273450619 (588h46m24.144533654s) [noise=true , address=77.90.31.16:4736 , piece_id=PSUNGGNF7EXLTZFXNUEWINIWFOS4X2IXP67RSKCIUYYUHU5YAH5Q, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.309222650 (588h46m24.180305685s) [noise=true , address=77.90.31.11:4555 , piece_id=ZSTA7TWF2SQYUYSB5GACGEENZVOFTN27GQX7IUUDKUYHIZN6K2XA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.404781931 (588h46m24.275864976s) [noise=true , address=45.91.116.6:4975 , piece_id=2I5I3A3DHEUL532UZROI5NPTOLL5WTA7VTMXSPOE5DFJFUCFVU2A, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.404812499 (588h46m24.275895535s) [seg_pos={0 0}, upload=516142]: Segment reached optimal threshold of 13 pieces. Dec 30 04:15:24.405079347 (588h46m24.276162422s) [noise=true , address=77.90.31.21:4878 , piece_id=NVWUX54T3OSYMCJIF62LFX5ZUHWKXF76OL5XCWBZKWIIXYQGAH4Q, seg_pos={0 0}, upload=516142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:47932->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:47932->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47932->77.90.31.21:4878: use of closed network connection Dec 30 04:15:24.405088976 (588h46m24.276172021s) [noise=true , address=77.90.31.21:4878 , piece_id=NVWUX54T3OSYMCJIF62LFX5ZUHWKXF76OL5XCWBZKWIIXYQGAH4Q, seg_pos={0 0}, upload=516142]: Long tail cancellation due to optimal threshold Dec 30 04:15:24.405393706 (588h46m24.276476742s) [noise=true , address=45.91.116.11:4119 , piece_id=5L25DQXXPYCMYIKBH5NRVA4V567NAUMMSM67FZ3I7A5YGA6V4G3Q, seg_pos={0 0}, upload=516142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 30 04:15:24.405400740 (588h46m24.276483775s) [noise=true , address=45.91.116.11:4119 , piece_id=5L25DQXXPYCMYIKBH5NRVA4V567NAUMMSM67FZ3I7A5YGA6V4G3Q, seg_pos={0 0}, upload=516142]: Long tail cancellation due to optimal threshold Dec 30 04:15:24.405611200 (588h46m24.276694235s) [noise=true , address=45.91.116.10:4081 , piece_id=SFMDWU7SYBSVWQXZHFD7GNLZPX4SRRARFBJ22WJA3D4MDGBMJSRA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: read tcp 188.166.225.170:50536->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:15:24.405633212 (588h46m24.276716267s) [noise=true , address=45.91.116.10:4081 , piece_id=SFMDWU7SYBSVWQXZHFD7GNLZPX4SRRARFBJ22WJA3D4MDGBMJSRA, seg_pos={0 0}, upload=516142]: Long tail cancellation due to optimal threshold Dec 30 04:15:24.405666535 (588h46m24.27674958s) [seg_pos={0 0}, upload=516142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:15:24.405857599 (588h46m24.276940645s) [upload=516142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:15:24.405867458 (588h46m24.276950503s) [upload=516142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:15:24.703060479 (588h46m24.574143514s) [upload=516142]: Done uploading segments... Dec 30 04:15:24.703086849 (588h46m24.574169884s) [upload=516142]: Upload finished. err: <nil> Duration: 5.866049578s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.X.......................X..XXX............XX..X.....X......] duration: 4.488609318s trace: https://jaeger.dev.storj.io/trace/408434392b868da5 FAIL (2025-12-30 04:16:03 UTC) orbiter warning: context deadline exceeded Duration: 4.488628766s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X............X..................] duration: 48.607501617s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:15:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 48.607533458s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.......XX.XX...XXXX...XXXX...X.X...XX..X.X..XX.......XX..X.X] duration: 30.003480591s trace: https://jaeger.dev.storj.io/trace/52a8d1ed6334dda7 FAIL (2025-12-30 04:15:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 2I5I3A3DHEUL532UZROI5NPTOLL5WTA7VTMXSPOE5DFJFUCFVU2A) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: PSUNGGNF7EXLTZFXNUEWINIWFOS4X2IXP67RSKCIUYYUHU5YAH5Q) context deadline exceeded ======LOGS====== 88h46m21.618666665s) [upload=516142]: Done starting upload Dec 30 04:15:21.747683661 (588h46m21.618766706s) [upload=516142]: Uploading segments... Dec 30 04:15:21.747801685 (588h46m21.61888472s) [upload=516142]: Got next segment. Inline: false Dec 30 04:15:21.747835690 (588h46m21.618918725s) [upload=516142]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:15:21.747840569 (588h46m21.618923604s) [upload=516142]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:15:22.002365677 (588h46m21.873448742s) [seg_pos={0 0}, upload=516142]: Begin upload segment... Dec 30 04:15:22.002426603 (588h46m21.873509648s) [seg_pos={0 0}, upload=516142]: Done begin upload segment. Dec 30 04:15:22.002498871 (588h46m21.873581906s) [noise=true , address=77.90.31.11:4555 , piece_id=ZSTA7TWF2SQYUYSB5GACGEENZVOFTN27GQX7IUUDKUYHIZN6K2XA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.002514872 (588h46m21.873597917s) [noise=true , address=145.79.227.5:4222 , piece_id=YMRSUP4MY2Q2W6MCISFDS3HQUWQN6HXEGMVZ54F5WU27HKQUACIQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.003130323 (588h46m21.874213368s) [noise=true , address=145.79.227.16:4909 , piece_id=QGRJWA2HW7SSJFY525DADV5TIDHAR3KFDMO3SNC3OAVBIYKGWUQQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.003452718 (588h46m21.874535763s) [noise=true , address=145.79.227.4:4187 , piece_id=ISJVYSCF5VDDKDROWFAS32EK7UTNOZKR7ULTXNKUHR2GS2JDI5XQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.003877076 (588h46m21.874960121s) [noise=true , address=45.91.116.10:4081 , piece_id=SFMDWU7SYBSVWQXZHFD7GNLZPX4SRRARFBJ22WJA3D4MDGBMJSRA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.004385774 (588h46m21.875468819s) [noise=true , address=145.79.227.3:4150 , piece_id=FPJ362RBQ4GAROMSUMPJ3TKFNUBEPX5XYYA6ATM6XGI6BNAUH3YQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.004840390 (588h46m21.875923425s) [noise=true , address=45.91.116.11:4119 , piece_id=5L25DQXXPYCMYIKBH5NRVA4V567NAUMMSM67FZ3I7A5YGA6V4G3Q, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.005316576 (588h46m21.876399611s) [noise=true , address=145.79.227.37:4566 , piece_id=OGJWRCWWMNUUYKOTOJTN2MDDVPEGO633J2H73CWATPGU5QJ3SKGA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.005970341 (588h46m21.877053386s) [noise=true , address=145.79.227.46:4601 , piece_id=3YNOCOUWS665FJD27U34OCSZQY7HMSZEXX3YPSWVM5S6T26NZ5FQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.006259603 (588h46m21.877342668s) [noise=true , address=145.79.227.15:4874 , piece_id=SCU6DJ7H5GA2HTE6XMMBOCA4JTMKDKEAHVHI4AP5UZMANMNDDUXQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.006558952 (588h46m21.877641987s) [noise=true , address=45.91.116.6:4975 , piece_id=2I5I3A3DHEUL532UZROI5NPTOLL5WTA7VTMXSPOE5DFJFUCFVU2A, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.007047553 (588h46m21.878130588s) [noise=true , address=77.90.31.21:4878 , piece_id=NVWUX54T3OSYMCJIF62LFX5ZUHWKXF76OL5XCWBZKWIIXYQGAH4Q, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.007620815 (588h46m21.87870386s) [noise=true , address=145.79.227.47:4265 , piece_id=7JTB2NTMH3CAT7ESWHFR2JRSXZGYKZI6CVHZFO3B44GJ5UIK6UMA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.008172646 (588h46m21.879255681s) [noise=true , address=145.79.227.20:4439 , piece_id=FBDGCBQ6NAURG4BNXWOVQNUABWLTBOFVA6UXF556LSEJF3JFLAYA, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.008749674 (588h46m21.879832749s) [noise=true , address=145.79.227.11:4729 , piece_id=OXWEVOXVQBJZNTIXOPJYGPPM4XYU7T5VW6PA4K7BCCTB4VDQV3KQ, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:22.008882637 (588h46m21.879965682s) [noise=true , address=77.90.31.16:4736 , piece_id=PSUNGGNF7EXLTZFXNUEWINIWFOS4X2IXP67RSKCIUYYUHU5YAH5Q, seg_pos={0 0}, upload=516142]: Uploading piece... Dec 30 04:15:23.711185336 (588h46m23.582268381s) [upload=516142]: Next returned nil segment Dec 30 04:15:23.711219642 (588h46m23.582302677s) [upload=516142]: Waiting for error group managing segments... Dec 30 04:15:23.993757614 (588h46m23.864840659s) [noise=true , address=145.79.227.3:4150 , piece_id=FPJ362RBQ4GAROMSUMPJ3TKFNUBEPX5XYYA6ATM6XGI6BNAUH3YQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:23.995291104 (588h46m23.866374149s) [noise=true , address=145.79.227.15:4874 , piece_id=SCU6DJ7H5GA2HTE6XMMBOCA4JTMKDKEAHVHI4AP5UZMANMNDDUXQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:23.997226881 (588h46m23.868309926s) [noise=true , address=145.79.227.16:4909 , piece_id=QGRJWA2HW7SSJFY525DADV5TIDHAR3KFDMO3SNC3OAVBIYKGWUQQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:23.997333694 (588h46m23.868416739s) [noise=true , address=145.79.227.20:4439 , piece_id=FBDGCBQ6NAURG4BNXWOVQNUABWLTBOFVA6UXF556LSEJF3JFLAYA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:23.999259902 (588h46m23.870342947s) [noise=true , address=145.79.227.11:4729 , piece_id=OXWEVOXVQBJZNTIXOPJYGPPM4XYU7T5VW6PA4K7BCCTB4VDQV3KQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.000377961 (588h46m23.871461007s) [noise=true , address=145.79.227.5:4222 , piece_id=YMRSUP4MY2Q2W6MCISFDS3HQUWQN6HXEGMVZ54F5WU27HKQUACIQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.010766804 (588h46m23.881849839s) [noise=true , address=145.79.227.46:4601 , piece_id=3YNOCOUWS665FJD27U34OCSZQY7HMSZEXX3YPSWVM5S6T26NZ5FQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.016615803 (588h46m23.887698848s) [noise=true , address=145.79.227.47:4265 , piece_id=7JTB2NTMH3CAT7ESWHFR2JRSXZGYKZI6CVHZFO3B44GJ5UIK6UMA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.021063222 (588h46m23.892146257s) [noise=true , address=145.79.227.4:4187 , piece_id=ISJVYSCF5VDDKDROWFAS32EK7UTNOZKR7ULTXNKUHR2GS2JDI5XQ, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.035011132 (588h46m23.906094177s) [noise=true , address=145.79.227.37:4566 , piece_id=OGJWRCWWMNUUYKOTOJTN2MDDVPEGO633J2H73CWATPGU5QJ3SKGA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.273450619 (588h46m24.144533654s) [noise=true , address=77.90.31.16:4736 , piece_id=PSUNGGNF7EXLTZFXNUEWINIWFOS4X2IXP67RSKCIUYYUHU5YAH5Q, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.309222650 (588h46m24.180305685s) [noise=true , address=77.90.31.11:4555 , piece_id=ZSTA7TWF2SQYUYSB5GACGEENZVOFTN27GQX7IUUDKUYHIZN6K2XA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.404781931 (588h46m24.275864976s) [noise=true , address=45.91.116.6:4975 , piece_id=2I5I3A3DHEUL532UZROI5NPTOLL5WTA7VTMXSPOE5DFJFUCFVU2A, seg_pos={0 0}, upload=516142]: Done uploading piece. err: <nil> Dec 30 04:15:24.404812499 (588h46m24.275895535s) [seg_pos={0 0}, upload=516142]: Segment reached optimal threshold of 13 pieces. Dec 30 04:15:24.405079347 (588h46m24.276162422s) [noise=true , address=77.90.31.21:4878 , piece_id=NVWUX54T3OSYMCJIF62LFX5ZUHWKXF76OL5XCWBZKWIIXYQGAH4Q, seg_pos={0 0}, upload=516142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:47932->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:47932->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47932->77.90.31.21:4878: use of closed network connection Dec 30 04:15:24.405088976 (588h46m24.276172021s) [noise=true , address=77.90.31.21:4878 , piece_id=NVWUX54T3OSYMCJIF62LFX5ZUHWKXF76OL5XCWBZKWIIXYQGAH4Q, seg_pos={0 0}, upload=516142]: Long tail cancellation due to optimal threshold Dec 30 04:15:24.405393706 (588h46m24.276476742s) [noise=true , address=45.91.116.11:4119 , piece_id=5L25DQXXPYCMYIKBH5NRVA4V567NAUMMSM67FZ3I7A5YGA6V4G3Q, seg_pos={0 0}, upload=516142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 30 04:15:24.405400740 (588h46m24.276483775s) [noise=true , address=45.91.116.11:4119 , piece_id=5L25DQXXPYCMYIKBH5NRVA4V567NAUMMSM67FZ3I7A5YGA6V4G3Q, seg_pos={0 0}, upload=516142]: Long tail cancellation due to optimal threshold Dec 30 04:15:24.405611200 (588h46m24.276694235s) [noise=true , address=45.91.116.10:4081 , piece_id=SFMDWU7SYBSVWQXZHFD7GNLZPX4SRRARFBJ22WJA3D4MDGBMJSRA, seg_pos={0 0}, upload=516142]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: read tcp 188.166.225.170:50536->45.91.116.10:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:15:24.405633212 (588h46m24.276716267s) [noise=true , address=45.91.116.10:4081 , piece_id=SFMDWU7SYBSVWQXZHFD7GNLZPX4SRRARFBJ22WJA3D4MDGBMJSRA, seg_pos={0 0}, upload=516142]: Long tail cancellation due to optimal threshold Dec 30 04:15:24.405666535 (588h46m24.27674958s) [seg_pos={0 0}, upload=516142]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:15:24.405857599 (588h46m24.276940645s) [upload=516142]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:15:24.405867458 (588h46m24.276950503s) [upload=516142]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:15:24.703060479 (588h46m24.574143514s) [upload=516142]: Done uploading segments... Dec 30 04:15:24.703086849 (588h46m24.574169884s) [upload=516142]: Upload finished. err: <nil> Duration: 30.00350676s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..X.......................X..XXX............XX..X.....X.....] duration: 4.233762167s trace: https://jaeger.dev.storj.io/trace/408434392b868da5 FAIL (2025-12-30 04:15:44 UTC) orbiter warning: context deadline exceeded Duration: 4.233790982s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...X.......................X..XXX............XX..X.....X....] duration: 3.431029455s trace: https://jaeger.dev.storj.io/trace/408434392b868da5 FAIL (2025-12-30 04:14:59 UTC) orbiter warning: context deadline exceeded Duration: 3.431069139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [........XX.XX...XXXX...XXXX...X.X...XX..X.X..XX.......XX..X.] duration: 7.460225753s trace: https://jaeger.dev.storj.io/trace/18961756bd5068e4 FAIL (2025-12-30 04:14:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: D4YA6GIUJHPHUKACUK4ZW4KAMZNTGREZKROVBXTKAH3VGH6JUSKQ) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: LIEU5ECXAGD7KPVA2BOGFQCX5EHQLODHCNMCKDAVW7LNNTFHZD4A) context deadline exceeded ======LOGS====== 2s) [upload=516122]: Got next segment. Inline: false Dec 30 04:14:04.924894552 (588h45m4.795977598s) [upload=516122]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:14:04.924900675 (588h45m4.79598372s) [upload=516122]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:14:05.171798308 (588h45m5.042881353s) [seg_pos={0 0}, upload=516122]: Begin upload segment... Dec 30 04:14:05.171856800 (588h45m5.042939845s) [seg_pos={0 0}, upload=516122]: Done begin upload segment. Dec 30 04:14:05.171900182 (588h45m5.042983227s) [noise=true , address=45.91.116.12:4227 , piece_id=MGWE4FIL4URMJKS2RQIV4MCMFB2TWHJZMU575YQU52JVNCZZ6NCQ, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.171906514 (588h45m5.042989549s) [noise=true , address=45.91.116.4:4940 , piece_id=P5UZD5RFGBG7IAA4RRXULPMCGULD25L5E5B4LCRTSDMEQO5DGEQQ, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.172000674 (588h45m5.043083719s) [noise=true , address=145.79.227.19:4079 , piece_id=2H6EJIEVDMHVT7IG7UK2O3TXHJNX33RRBDA7PEMEDDEKOMHQYQDQ, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.172565198 (588h45m5.043648243s) [noise=true , address=145.79.227.9:4367 , piece_id=MZS6GFM5P6VAA67FJFFIYUVUG35CYDV2FBWSJXFZTAOFCOXYCOCA, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.172849910 (588h45m5.043932955s) [noise=true , address=145.79.227.5:4222 , piece_id=HUWBHNUHUVIKUY56OKP4TSZGFOTXGDLUACYAEU6KMZOQH7HJYINQ, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.173122690 (588h45m5.044205735s) [noise=true , address=45.91.116.6:4975 , piece_id=SH672GWXVBRKG6RPDJPLFXKXBX7QSSHVSQRG7A7GKR3G3QTKRXJQ, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.173425457 (588h45m5.044508492s) [noise=true , address=145.79.227.4:4185 , piece_id=QT6DLGLBLFEULCN73GPHGMQXLLOIPQ6VY5YWYTMFDJ6ALF2MPW5Q, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.173768310 (588h45m5.044851355s) [noise=true , address=77.90.31.14:4661 , piece_id=LIEU5ECXAGD7KPVA2BOGFQCX5EHQLODHCNMCKDAVW7LNNTFHZD4A, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.174117475 (588h45m5.04520051s) [noise=true , address=45.91.116.11:4117 , piece_id=3SN4M5F6SBHZNSP7XEAEB64A3PFNG5KZPPWDGHFJ2JEXSESD4T2Q, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.174650840 (588h45m5.045733875s) [noise=true , address=145.79.227.47:4266 , piece_id=5SHUL3ADIVIF2AL6SCWQSRDOQBXCWC6CQBXCTZHKKWOWC7FTVWBA, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.176609440 (588h45m5.047692485s) [noise=true , address=145.79.227.11:4730 , piece_id=PSCLQOTOGGQ7IQUDYHGD3WXTF3KBN7E75YCWM6ISWT5PEFNARLGA, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.178343001 (588h45m5.049426046s) [noise=true , address=45.91.116.5:4299 , piece_id=W7IYBLJP7YEZXXAIBLTFUJGL62TBNILFJLS3GJ5BYMWYLX4GZGTQ, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.179804604 (588h45m5.050887649s) [noise=true , address=45.91.116.7:4012 , piece_id=NO4O7CH6PMBOKIMWK7BL3NEYQUL43UVWGYDDVGMFZG52JGANJMSA, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.181472350 (588h45m5.052555385s) [noise=true , address=45.91.116.9:4262 , piece_id=D4YA6GIUJHPHUKACUK4ZW4KAMZNTGREZKROVBXTKAH3VGH6JUSKQ, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.182818474 (588h45m5.053901519s) [noise=true , address=145.79.227.45:4231 , piece_id=UX5EVCVKDT5YC6V4UAWBJ5DQGZQ6VTIPC33O3WM43QGGME2TH5VQ, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.184043446 (588h45m5.055126491s) [noise=true , address=77.90.31.12:4589 , piece_id=7PG6N457YYQFA5DV5YHM3ZKUEWLSEYIVMGI3RGRG34J5RZBFZWKA, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:06.877735819 (588h45m6.748818864s) [upload=516122]: Next returned nil segment Dec 30 04:14:06.877753502 (588h45m6.748836547s) [upload=516122]: Waiting for error group managing segments... Dec 30 04:14:07.160016442 (588h45m7.031099487s) [noise=true , address=145.79.227.5:4222 , piece_id=HUWBHNUHUVIKUY56OKP4TSZGFOTXGDLUACYAEU6KMZOQH7HJYINQ, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.166993037 (588h45m7.038076082s) [noise=true , address=145.79.227.19:4079 , piece_id=2H6EJIEVDMHVT7IG7UK2O3TXHJNX33RRBDA7PEMEDDEKOMHQYQDQ, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.170747497 (588h45m7.041830542s) [noise=true , address=145.79.227.45:4231 , piece_id=UX5EVCVKDT5YC6V4UAWBJ5DQGZQ6VTIPC33O3WM43QGGME2TH5VQ, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.181279543 (588h45m7.052362588s) [noise=true , address=145.79.227.4:4185 , piece_id=QT6DLGLBLFEULCN73GPHGMQXLLOIPQ6VY5YWYTMFDJ6ALF2MPW5Q, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.184376460 (588h45m7.055459505s) [noise=true , address=145.79.227.9:4367 , piece_id=MZS6GFM5P6VAA67FJFFIYUVUG35CYDV2FBWSJXFZTAOFCOXYCOCA, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.185158179 (588h45m7.056241224s) [noise=true , address=145.79.227.11:4730 , piece_id=PSCLQOTOGGQ7IQUDYHGD3WXTF3KBN7E75YCWM6ISWT5PEFNARLGA, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.196811430 (588h45m7.067894475s) [noise=true , address=145.79.227.47:4266 , piece_id=5SHUL3ADIVIF2AL6SCWQSRDOQBXCWC6CQBXCTZHKKWOWC7FTVWBA, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.475963875 (588h45m7.34704692s) [noise=true , address=77.90.31.14:4661 , piece_id=LIEU5ECXAGD7KPVA2BOGFQCX5EHQLODHCNMCKDAVW7LNNTFHZD4A, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.506999035 (588h45m7.37808208s) [noise=true , address=45.91.116.9:4262 , piece_id=D4YA6GIUJHPHUKACUK4ZW4KAMZNTGREZKROVBXTKAH3VGH6JUSKQ, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.520558927 (588h45m7.391641972s) [noise=true , address=45.91.116.4:4940 , piece_id=P5UZD5RFGBG7IAA4RRXULPMCGULD25L5E5B4LCRTSDMEQO5DGEQQ, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.563815220 (588h45m7.434898255s) [noise=true , address=45.91.116.5:4299 , piece_id=W7IYBLJP7YEZXXAIBLTFUJGL62TBNILFJLS3GJ5BYMWYLX4GZGTQ, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.570998619 (588h45m7.442081654s) [noise=true , address=77.90.31.12:4589 , piece_id=7PG6N457YYQFA5DV5YHM3ZKUEWLSEYIVMGI3RGRG34J5RZBFZWKA, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.614479779 (588h45m7.485562814s) [noise=true , address=45.91.116.12:4227 , piece_id=MGWE4FIL4URMJKS2RQIV4MCMFB2TWHJZMU575YQU52JVNCZZ6NCQ, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.614499837 (588h45m7.485582872s) [seg_pos={0 0}, upload=516122]: Segment reached optimal threshold of 13 pieces. Dec 30 04:14:07.614780231 (588h45m7.485863276s) [noise=true , address=45.91.116.6:4975 , piece_id=SH672GWXVBRKG6RPDJPLFXKXBX7QSSHVSQRG7A7GKR3G3QTKRXJQ, seg_pos={0 0}, upload=516122]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:43470->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:43470->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43470->45.91.116.6:4975: use of closed network connection Dec 30 04:14:07.614792956 (588h45m7.485875991s) [noise=true , address=45.91.116.6:4975 , piece_id=SH672GWXVBRKG6RPDJPLFXKXBX7QSSHVSQRG7A7GKR3G3QTKRXJQ, seg_pos={0 0}, upload=516122]: Long tail cancellation due to optimal threshold Dec 30 04:14:07.614929867 (588h45m7.486012912s) [noise=true , address=45.91.116.7:4012 , piece_id=NO4O7CH6PMBOKIMWK7BL3NEYQUL43UVWGYDDVGMFZG52JGANJMSA, seg_pos={0 0}, upload=516122]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:49600->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:49600->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49600->45.91.116.7:4012: use of closed network connection Dec 30 04:14:07.614938913 (588h45m7.486021959s) [noise=true , address=45.91.116.7:4012 , piece_id=NO4O7CH6PMBOKIMWK7BL3NEYQUL43UVWGYDDVGMFZG52JGANJMSA, seg_pos={0 0}, upload=516122]: Long tail cancellation due to optimal threshold Dec 30 04:14:07.615052299 (588h45m7.486135344s) [noise=true , address=45.91.116.11:4117 , piece_id=3SN4M5F6SBHZNSP7XEAEB64A3PFNG5KZPPWDGHFJ2JEXSESD4T2Q, seg_pos={0 0}, upload=516122]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: read tcp 188.166.225.170:56806->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:14:07.615073420 (588h45m7.486156465s) [noise=true , address=45.91.116.11:4117 , piece_id=3SN4M5F6SBHZNSP7XEAEB64A3PFNG5KZPPWDGHFJ2JEXSESD4T2Q, seg_pos={0 0}, upload=516122]: Long tail cancellation due to optimal threshold Dec 30 04:14:07.615101153 (588h45m7.486184198s) [seg_pos={0 0}, upload=516122]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:14:07.615228114 (588h45m7.486311159s) [upload=516122]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:14:07.615366578 (588h45m7.486449653s) [upload=516122]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:14:07.909603108 (588h45m7.780686163s) [upload=516122]: Done uploading segments... Dec 30 04:14:07.909628527 (588h45m7.780711562s) [upload=516122]: Upload finished. err: <nil> Duration: 7.460242895s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X........XX.XX...XXXX...XXXX...X.X...XX..X.X..XX.......XX..X] duration: 30.003441423s trace: https://jaeger.dev.storj.io/trace/18961756bd5068e4 FAIL (2025-12-30 04:14:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: D4YA6GIUJHPHUKACUK4ZW4KAMZNTGREZKROVBXTKAH3VGH6JUSKQ) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: LIEU5ECXAGD7KPVA2BOGFQCX5EHQLODHCNMCKDAVW7LNNTFHZD4A) context deadline exceeded ======LOGS====== 2s) [upload=516122]: Got next segment. Inline: false Dec 30 04:14:04.924894552 (588h45m4.795977598s) [upload=516122]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:14:04.924900675 (588h45m4.79598372s) [upload=516122]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:14:05.171798308 (588h45m5.042881353s) [seg_pos={0 0}, upload=516122]: Begin upload segment... Dec 30 04:14:05.171856800 (588h45m5.042939845s) [seg_pos={0 0}, upload=516122]: Done begin upload segment. Dec 30 04:14:05.171900182 (588h45m5.042983227s) [noise=true , address=45.91.116.12:4227 , piece_id=MGWE4FIL4URMJKS2RQIV4MCMFB2TWHJZMU575YQU52JVNCZZ6NCQ, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.171906514 (588h45m5.042989549s) [noise=true , address=45.91.116.4:4940 , piece_id=P5UZD5RFGBG7IAA4RRXULPMCGULD25L5E5B4LCRTSDMEQO5DGEQQ, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.172000674 (588h45m5.043083719s) [noise=true , address=145.79.227.19:4079 , piece_id=2H6EJIEVDMHVT7IG7UK2O3TXHJNX33RRBDA7PEMEDDEKOMHQYQDQ, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.172565198 (588h45m5.043648243s) [noise=true , address=145.79.227.9:4367 , piece_id=MZS6GFM5P6VAA67FJFFIYUVUG35CYDV2FBWSJXFZTAOFCOXYCOCA, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.172849910 (588h45m5.043932955s) [noise=true , address=145.79.227.5:4222 , piece_id=HUWBHNUHUVIKUY56OKP4TSZGFOTXGDLUACYAEU6KMZOQH7HJYINQ, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.173122690 (588h45m5.044205735s) [noise=true , address=45.91.116.6:4975 , piece_id=SH672GWXVBRKG6RPDJPLFXKXBX7QSSHVSQRG7A7GKR3G3QTKRXJQ, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.173425457 (588h45m5.044508492s) [noise=true , address=145.79.227.4:4185 , piece_id=QT6DLGLBLFEULCN73GPHGMQXLLOIPQ6VY5YWYTMFDJ6ALF2MPW5Q, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.173768310 (588h45m5.044851355s) [noise=true , address=77.90.31.14:4661 , piece_id=LIEU5ECXAGD7KPVA2BOGFQCX5EHQLODHCNMCKDAVW7LNNTFHZD4A, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.174117475 (588h45m5.04520051s) [noise=true , address=45.91.116.11:4117 , piece_id=3SN4M5F6SBHZNSP7XEAEB64A3PFNG5KZPPWDGHFJ2JEXSESD4T2Q, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.174650840 (588h45m5.045733875s) [noise=true , address=145.79.227.47:4266 , piece_id=5SHUL3ADIVIF2AL6SCWQSRDOQBXCWC6CQBXCTZHKKWOWC7FTVWBA, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.176609440 (588h45m5.047692485s) [noise=true , address=145.79.227.11:4730 , piece_id=PSCLQOTOGGQ7IQUDYHGD3WXTF3KBN7E75YCWM6ISWT5PEFNARLGA, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.178343001 (588h45m5.049426046s) [noise=true , address=45.91.116.5:4299 , piece_id=W7IYBLJP7YEZXXAIBLTFUJGL62TBNILFJLS3GJ5BYMWYLX4GZGTQ, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.179804604 (588h45m5.050887649s) [noise=true , address=45.91.116.7:4012 , piece_id=NO4O7CH6PMBOKIMWK7BL3NEYQUL43UVWGYDDVGMFZG52JGANJMSA, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.181472350 (588h45m5.052555385s) [noise=true , address=45.91.116.9:4262 , piece_id=D4YA6GIUJHPHUKACUK4ZW4KAMZNTGREZKROVBXTKAH3VGH6JUSKQ, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.182818474 (588h45m5.053901519s) [noise=true , address=145.79.227.45:4231 , piece_id=UX5EVCVKDT5YC6V4UAWBJ5DQGZQ6VTIPC33O3WM43QGGME2TH5VQ, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:05.184043446 (588h45m5.055126491s) [noise=true , address=77.90.31.12:4589 , piece_id=7PG6N457YYQFA5DV5YHM3ZKUEWLSEYIVMGI3RGRG34J5RZBFZWKA, seg_pos={0 0}, upload=516122]: Uploading piece... Dec 30 04:14:06.877735819 (588h45m6.748818864s) [upload=516122]: Next returned nil segment Dec 30 04:14:06.877753502 (588h45m6.748836547s) [upload=516122]: Waiting for error group managing segments... Dec 30 04:14:07.160016442 (588h45m7.031099487s) [noise=true , address=145.79.227.5:4222 , piece_id=HUWBHNUHUVIKUY56OKP4TSZGFOTXGDLUACYAEU6KMZOQH7HJYINQ, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.166993037 (588h45m7.038076082s) [noise=true , address=145.79.227.19:4079 , piece_id=2H6EJIEVDMHVT7IG7UK2O3TXHJNX33RRBDA7PEMEDDEKOMHQYQDQ, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.170747497 (588h45m7.041830542s) [noise=true , address=145.79.227.45:4231 , piece_id=UX5EVCVKDT5YC6V4UAWBJ5DQGZQ6VTIPC33O3WM43QGGME2TH5VQ, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.181279543 (588h45m7.052362588s) [noise=true , address=145.79.227.4:4185 , piece_id=QT6DLGLBLFEULCN73GPHGMQXLLOIPQ6VY5YWYTMFDJ6ALF2MPW5Q, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.184376460 (588h45m7.055459505s) [noise=true , address=145.79.227.9:4367 , piece_id=MZS6GFM5P6VAA67FJFFIYUVUG35CYDV2FBWSJXFZTAOFCOXYCOCA, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.185158179 (588h45m7.056241224s) [noise=true , address=145.79.227.11:4730 , piece_id=PSCLQOTOGGQ7IQUDYHGD3WXTF3KBN7E75YCWM6ISWT5PEFNARLGA, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.196811430 (588h45m7.067894475s) [noise=true , address=145.79.227.47:4266 , piece_id=5SHUL3ADIVIF2AL6SCWQSRDOQBXCWC6CQBXCTZHKKWOWC7FTVWBA, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.475963875 (588h45m7.34704692s) [noise=true , address=77.90.31.14:4661 , piece_id=LIEU5ECXAGD7KPVA2BOGFQCX5EHQLODHCNMCKDAVW7LNNTFHZD4A, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.506999035 (588h45m7.37808208s) [noise=true , address=45.91.116.9:4262 , piece_id=D4YA6GIUJHPHUKACUK4ZW4KAMZNTGREZKROVBXTKAH3VGH6JUSKQ, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.520558927 (588h45m7.391641972s) [noise=true , address=45.91.116.4:4940 , piece_id=P5UZD5RFGBG7IAA4RRXULPMCGULD25L5E5B4LCRTSDMEQO5DGEQQ, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.563815220 (588h45m7.434898255s) [noise=true , address=45.91.116.5:4299 , piece_id=W7IYBLJP7YEZXXAIBLTFUJGL62TBNILFJLS3GJ5BYMWYLX4GZGTQ, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.570998619 (588h45m7.442081654s) [noise=true , address=77.90.31.12:4589 , piece_id=7PG6N457YYQFA5DV5YHM3ZKUEWLSEYIVMGI3RGRG34J5RZBFZWKA, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.614479779 (588h45m7.485562814s) [noise=true , address=45.91.116.12:4227 , piece_id=MGWE4FIL4URMJKS2RQIV4MCMFB2TWHJZMU575YQU52JVNCZZ6NCQ, seg_pos={0 0}, upload=516122]: Done uploading piece. err: <nil> Dec 30 04:14:07.614499837 (588h45m7.485582872s) [seg_pos={0 0}, upload=516122]: Segment reached optimal threshold of 13 pieces. Dec 30 04:14:07.614780231 (588h45m7.485863276s) [noise=true , address=45.91.116.6:4975 , piece_id=SH672GWXVBRKG6RPDJPLFXKXBX7QSSHVSQRG7A7GKR3G3QTKRXJQ, seg_pos={0 0}, upload=516122]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: write tcp 188.166.225.170:43470->45.91.116.6:4975: use of closed network connection; write tcp 188.166.225.170:43470->45.91.116.6:4975: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43470->45.91.116.6:4975: use of closed network connection Dec 30 04:14:07.614792956 (588h45m7.485875991s) [noise=true , address=45.91.116.6:4975 , piece_id=SH672GWXVBRKG6RPDJPLFXKXBX7QSSHVSQRG7A7GKR3G3QTKRXJQ, seg_pos={0 0}, upload=516122]: Long tail cancellation due to optimal threshold Dec 30 04:14:07.614929867 (588h45m7.486012912s) [noise=true , address=45.91.116.7:4012 , piece_id=NO4O7CH6PMBOKIMWK7BL3NEYQUL43UVWGYDDVGMFZG52JGANJMSA, seg_pos={0 0}, upload=516122]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: write tcp 188.166.225.170:49600->45.91.116.7:4012: use of closed network connection; write tcp 188.166.225.170:49600->45.91.116.7:4012: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49600->45.91.116.7:4012: use of closed network connection Dec 30 04:14:07.614938913 (588h45m7.486021959s) [noise=true , address=45.91.116.7:4012 , piece_id=NO4O7CH6PMBOKIMWK7BL3NEYQUL43UVWGYDDVGMFZG52JGANJMSA, seg_pos={0 0}, upload=516122]: Long tail cancellation due to optimal threshold Dec 30 04:14:07.615052299 (588h45m7.486135344s) [noise=true , address=45.91.116.11:4117 , piece_id=3SN4M5F6SBHZNSP7XEAEB64A3PFNG5KZPPWDGHFJ2JEXSESD4T2Q, seg_pos={0 0}, upload=516122]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: read tcp 188.166.225.170:56806->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 30 04:14:07.615073420 (588h45m7.486156465s) [noise=true , address=45.91.116.11:4117 , piece_id=3SN4M5F6SBHZNSP7XEAEB64A3PFNG5KZPPWDGHFJ2JEXSESD4T2Q, seg_pos={0 0}, upload=516122]: Long tail cancellation due to optimal threshold Dec 30 04:14:07.615101153 (588h45m7.486184198s) [seg_pos={0 0}, upload=516122]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:14:07.615228114 (588h45m7.486311159s) [upload=516122]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:14:07.615366578 (588h45m7.486449653s) [upload=516122]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:14:07.909603108 (588h45m7.780686163s) [upload=516122]: Done uploading segments... Dec 30 04:14:07.909628527 (588h45m7.780711562s) [upload=516122]: Upload finished. err: <nil> Duration: 30.003473774s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X............X.................] duration: 15.459376683s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:14:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 15.459405478s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X............X................] duration: 17.90140475s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:14:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 17.901423985s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X...X.......................X..XXX............XX..X.....X...] duration: 3.440479842s trace: https://jaeger.dev.storj.io/trace/408434392b868da5 FAIL (2025-12-30 04:14:12 UTC) orbiter warning: context deadline exceeded Duration: 3.440512384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X............X...............] duration: 46.237530118s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:13:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 46.237564684s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [XX...X.......................X..XXX............XX..X.....X..] duration: 3.501868524s trace: https://jaeger.dev.storj.io/trace/408434392b868da5 FAIL (2025-12-30 04:13:47 UTC) orbiter warning: context deadline exceeded Duration: 3.50189304s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX........XX.XX...XXXX...XXXX...X.X...XX..X.X..XX.......XX..] duration: 15.908759649s trace: https://jaeger.dev.storj.io/trace/d7820782ab9ff69 FAIL (2025-12-30 04:13:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: 7IBPBSMR36JFMDXEIHHQHJGCNR44IQVRSK2LSIQIHK2ITYW2QTFA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: HF672QJYI2LHHAIFL4KIH3KPYOJSAKSOKEMFAD3VRI6SDIFWWUUQ) context deadline exceeded ======LOGS====== Dec 30 04:12:26.206421831 (588h43m26.077504866s) [upload=516094]: Starting upload Dec 30 04:12:26.206542230 (588h43m26.077625275s) [upload=516094]: Done starting upload Dec 30 04:12:26.206613877 (588h43m26.077696922s) [upload=516094]: Uploading segments... Dec 30 04:12:26.206825821 (588h43m26.077908866s) [upload=516094]: Got next segment. Inline: false Dec 30 04:12:26.206892287 (588h43m26.077975333s) [upload=516094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:12:26.206899070 (588h43m26.077982115s) [upload=516094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:12:26.455180628 (588h43m26.326263663s) [seg_pos={0 0}, upload=516094]: Begin upload segment... Dec 30 04:12:26.455236104 (588h43m26.326319139s) [seg_pos={0 0}, upload=516094]: Done begin upload segment. Dec 30 04:12:26.455289436 (588h43m26.326372481s) [noise=true , address=145.79.227.4:4185 , piece_id=VU2W4WHRCNFDB6DMQFJOOWA7RWLHMON36WCJEYBIOK76TIJDGY3A, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.455306849 (588h43m26.326389884s) [noise=true , address=45.91.116.9:4262 , piece_id=7IBPBSMR36JFMDXEIHHQHJGCNR44IQVRSK2LSIQIHK2ITYW2QTFA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.455800058 (588h43m26.326883093s) [noise=true , address=145.79.227.8:4329 , piece_id=5PBTZTK3WCEYL7UNXDNK7ED5UZSBC7DQ56EKRFF3AKX6EM33C55A, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.456082475 (588h43m26.32716552s) [noise=true , address=45.91.116.4:4940 , piece_id=F2QGTNWOE3E4XDFYPR6D5YLEYB7X74T6TGT4TUZ3XXN6WETBPIMA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.456505972 (588h43m26.327589007s) [noise=true , address=145.79.227.7:4294 , piece_id=VQXBFHDIMOA4AR2V5IU4D7WHGCTQOVQFS5FFVSMHIEL6RMXBUVSA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.456939117 (588h43m26.328022162s) [noise=true , address=145.79.227.9:4365 , piece_id=RC6CODBHZPOLTXYSSATCVYZGBHR6Q6BXPJVDOFRCQBRO4W2BGKGA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.457286819 (588h43m26.328369854s) [noise=true , address=145.79.227.42:4123 , piece_id=7UDGXR62LWGGWAKKSHEUCR5FGYEGZJMU3MW4AS3O7RPHCRPKS4OA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.458782647 (588h43m26.329865692s) [noise=true , address=145.79.227.17:4971 , piece_id=2HZERGO7D2LCLXTTFVJRHL6REXQQR5FTIQX2QRZYWSR3BOOUEBFA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.460064408 (588h43m26.331147453s) [noise=true , address=145.79.227.41:4086 , piece_id=5ESIXVVZ5R6TZ7KC7OX7KURQ2BGHWQVLR7IOB6KHEUDAYXSOS7ZA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.461325209 (588h43m26.332408254s) [noise=true , address=77.90.31.15:4700 , piece_id=HF672QJYI2LHHAIFL4KIH3KPYOJSAKSOKEMFAD3VRI6SDIFWWUUQ, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.463047268 (588h43m26.334130313s) [noise=true , address=145.79.227.6:4258 , piece_id=VNQTOHC6AXNPTVVPOGJ5DYSGEUKRMFAY2GG3LJKOXYDEMZEKIIVQ, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.464893033 (588h43m26.335976068s) [noise=true , address=45.91.116.11:4120 , piece_id=PPPFXRPE6Q7X2QF3HHOT6Y6JSIACBRWUSNYJIBFRAGYIVPKEVHYA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.466197426 (588h43m26.337280471s) [noise=true , address=145.79.227.23:4545 , piece_id=5NVNPRC6MTKCYNXQAP4RVX66HKGAHFYZ3O4T6TBMMNXRPV5SCQSQ, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.466924340 (588h43m26.338007376s) [noise=true , address=145.79.227.35:4481 , piece_id=KCSQ4D4QXCRDBEVQ23NL2CUSVKINM6VEO5AJB7D64NJ5UOKR6KMA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.468159393 (588h43m26.339242428s) [noise=true , address=145.79.227.33:4871 , piece_id=VO6WL36M2PEQ4CH3TB5TSNQXMCHUW2CYSINHZ6UTP4OLUIZWXMHQ, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.471152222 (588h43m26.342235257s) [noise=true , address=145.79.227.10:4402 , piece_id=TZ2MT3M2VZAKKLRX7MX3RYBDKGASGMVJ6E3Z3L7TRR2MJGGKGWGQ, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:28.064015810 (588h43m27.935098855s) [upload=516094]: Next returned nil segment Dec 30 04:12:28.064028674 (588h43m27.935111709s) [upload=516094]: Waiting for error group managing segments... Dec 30 04:12:28.433157354 (588h43m28.304240389s) [noise=true , address=145.79.227.6:4258 , piece_id=VNQTOHC6AXNPTVVPOGJ5DYSGEUKRMFAY2GG3LJKOXYDEMZEKIIVQ, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.439536271 (588h43m28.310619316s) [noise=true , address=145.79.227.7:4294 , piece_id=VQXBFHDIMOA4AR2V5IU4D7WHGCTQOVQFS5FFVSMHIEL6RMXBUVSA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.446052540 (588h43m28.317135575s) [noise=true , address=145.79.227.8:4329 , piece_id=5PBTZTK3WCEYL7UNXDNK7ED5UZSBC7DQ56EKRFF3AKX6EM33C55A, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.449775599 (588h43m28.320858645s) [noise=true , address=145.79.227.41:4086 , piece_id=5ESIXVVZ5R6TZ7KC7OX7KURQ2BGHWQVLR7IOB6KHEUDAYXSOS7ZA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.450978080 (588h43m28.322061115s) [noise=true , address=145.79.227.9:4365 , piece_id=RC6CODBHZPOLTXYSSATCVYZGBHR6Q6BXPJVDOFRCQBRO4W2BGKGA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.454408401 (588h43m28.325491436s) [noise=true , address=145.79.227.23:4545 , piece_id=5NVNPRC6MTKCYNXQAP4RVX66HKGAHFYZ3O4T6TBMMNXRPV5SCQSQ, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.458105311 (588h43m28.329188346s) [noise=true , address=145.79.227.17:4971 , piece_id=2HZERGO7D2LCLXTTFVJRHL6REXQQR5FTIQX2QRZYWSR3BOOUEBFA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.461820194 (588h43m28.332903229s) [noise=true , address=145.79.227.10:4402 , piece_id=TZ2MT3M2VZAKKLRX7MX3RYBDKGASGMVJ6E3Z3L7TRR2MJGGKGWGQ, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.462558411 (588h43m28.333641446s) [noise=true , address=145.79.227.4:4185 , piece_id=VU2W4WHRCNFDB6DMQFJOOWA7RWLHMON36WCJEYBIOK76TIJDGY3A, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.473689927 (588h43m28.344772963s) [noise=true , address=145.79.227.42:4123 , piece_id=7UDGXR62LWGGWAKKSHEUCR5FGYEGZJMU3MW4AS3O7RPHCRPKS4OA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.724530027 (588h43m28.595613073s) [noise=true , address=45.91.116.9:4262 , piece_id=7IBPBSMR36JFMDXEIHHQHJGCNR44IQVRSK2LSIQIHK2ITYW2QTFA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.741946263 (588h43m28.613029308s) [noise=true , address=45.91.116.11:4120 , piece_id=PPPFXRPE6Q7X2QF3HHOT6Y6JSIACBRWUSNYJIBFRAGYIVPKEVHYA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:29.350263495 (588h43m29.22134653s) [noise=true , address=77.90.31.15:4700 , piece_id=HF672QJYI2LHHAIFL4KIH3KPYOJSAKSOKEMFAD3VRI6SDIFWWUUQ, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:29.350333468 (588h43m29.221416513s) [seg_pos={0 0}, upload=516094]: Segment reached optimal threshold of 13 pieces. Dec 30 04:12:29.350527718 (588h43m29.221610753s) [noise=true , address=145.79.227.35:4481 , piece_id=KCSQ4D4QXCRDBEVQ23NL2CUSVKINM6VEO5AJB7D64NJ5UOKR6KMA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 04:12:29.350537647 (588h43m29.221620682s) [noise=true , address=145.79.227.35:4481 , piece_id=KCSQ4D4QXCRDBEVQ23NL2CUSVKINM6VEO5AJB7D64NJ5UOKR6KMA, seg_pos={0 0}, upload=516094]: Long tail cancellation due to optimal threshold Dec 30 04:12:29.350689315 (588h43m29.22177236s) [noise=true , address=145.79.227.33:4871 , piece_id=VO6WL36M2PEQ4CH3TB5TSNQXMCHUW2CYSINHZ6UTP4OLUIZWXMHQ, seg_pos={0 0}, upload=516094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 30 04:12:29.350713472 (588h43m29.221796517s) [noise=true , address=145.79.227.33:4871 , piece_id=VO6WL36M2PEQ4CH3TB5TSNQXMCHUW2CYSINHZ6UTP4OLUIZWXMHQ, seg_pos={0 0}, upload=516094]: Long tail cancellation due to optimal threshold Dec 30 04:12:29.350849250 (588h43m29.221932295s) [noise=true , address=45.91.116.4:4940 , piece_id=F2QGTNWOE3E4XDFYPR6D5YLEYB7X74T6TGT4TUZ3XXN6WETBPIMA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 04:12:29.350863497 (588h43m29.221946552s) [noise=true , address=45.91.116.4:4940 , piece_id=F2QGTNWOE3E4XDFYPR6D5YLEYB7X74T6TGT4TUZ3XXN6WETBPIMA, seg_pos={0 0}, upload=516094]: Long tail cancellation due to optimal threshold Dec 30 04:12:29.350915526 (588h43m29.221998562s) [seg_pos={0 0}, upload=516094]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:12:29.350994016 (588h43m29.222077061s) [upload=516094]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:12:29.350999276 (588h43m29.222082311s) [upload=516094]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:12:29.638767606 (588h43m29.509850641s) [upload=516094]: Done uploading segments... Dec 30 04:12:29.638786313 (588h43m29.509869348s) [upload=516094]: Upload finished. err: <nil> Duration: 15.908795487s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXX........XX.XX...XXXX...XXXX...X.X...XX..X.X..XX.......XX.] duration: 5.999134938s trace: https://jaeger.dev.storj.io/trace/d7820782ab9ff69 FAIL (2025-12-30 04:13:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: 7IBPBSMR36JFMDXEIHHQHJGCNR44IQVRSK2LSIQIHK2ITYW2QTFA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: HF672QJYI2LHHAIFL4KIH3KPYOJSAKSOKEMFAD3VRI6SDIFWWUUQ) context deadline exceeded ======LOGS====== Dec 30 04:12:26.206421831 (588h43m26.077504866s) [upload=516094]: Starting upload Dec 30 04:12:26.206542230 (588h43m26.077625275s) [upload=516094]: Done starting upload Dec 30 04:12:26.206613877 (588h43m26.077696922s) [upload=516094]: Uploading segments... Dec 30 04:12:26.206825821 (588h43m26.077908866s) [upload=516094]: Got next segment. Inline: false Dec 30 04:12:26.206892287 (588h43m26.077975333s) [upload=516094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:12:26.206899070 (588h43m26.077982115s) [upload=516094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:12:26.455180628 (588h43m26.326263663s) [seg_pos={0 0}, upload=516094]: Begin upload segment... Dec 30 04:12:26.455236104 (588h43m26.326319139s) [seg_pos={0 0}, upload=516094]: Done begin upload segment. Dec 30 04:12:26.455289436 (588h43m26.326372481s) [noise=true , address=145.79.227.4:4185 , piece_id=VU2W4WHRCNFDB6DMQFJOOWA7RWLHMON36WCJEYBIOK76TIJDGY3A, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.455306849 (588h43m26.326389884s) [noise=true , address=45.91.116.9:4262 , piece_id=7IBPBSMR36JFMDXEIHHQHJGCNR44IQVRSK2LSIQIHK2ITYW2QTFA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.455800058 (588h43m26.326883093s) [noise=true , address=145.79.227.8:4329 , piece_id=5PBTZTK3WCEYL7UNXDNK7ED5UZSBC7DQ56EKRFF3AKX6EM33C55A, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.456082475 (588h43m26.32716552s) [noise=true , address=45.91.116.4:4940 , piece_id=F2QGTNWOE3E4XDFYPR6D5YLEYB7X74T6TGT4TUZ3XXN6WETBPIMA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.456505972 (588h43m26.327589007s) [noise=true , address=145.79.227.7:4294 , piece_id=VQXBFHDIMOA4AR2V5IU4D7WHGCTQOVQFS5FFVSMHIEL6RMXBUVSA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.456939117 (588h43m26.328022162s) [noise=true , address=145.79.227.9:4365 , piece_id=RC6CODBHZPOLTXYSSATCVYZGBHR6Q6BXPJVDOFRCQBRO4W2BGKGA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.457286819 (588h43m26.328369854s) [noise=true , address=145.79.227.42:4123 , piece_id=7UDGXR62LWGGWAKKSHEUCR5FGYEGZJMU3MW4AS3O7RPHCRPKS4OA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.458782647 (588h43m26.329865692s) [noise=true , address=145.79.227.17:4971 , piece_id=2HZERGO7D2LCLXTTFVJRHL6REXQQR5FTIQX2QRZYWSR3BOOUEBFA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.460064408 (588h43m26.331147453s) [noise=true , address=145.79.227.41:4086 , piece_id=5ESIXVVZ5R6TZ7KC7OX7KURQ2BGHWQVLR7IOB6KHEUDAYXSOS7ZA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.461325209 (588h43m26.332408254s) [noise=true , address=77.90.31.15:4700 , piece_id=HF672QJYI2LHHAIFL4KIH3KPYOJSAKSOKEMFAD3VRI6SDIFWWUUQ, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.463047268 (588h43m26.334130313s) [noise=true , address=145.79.227.6:4258 , piece_id=VNQTOHC6AXNPTVVPOGJ5DYSGEUKRMFAY2GG3LJKOXYDEMZEKIIVQ, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.464893033 (588h43m26.335976068s) [noise=true , address=45.91.116.11:4120 , piece_id=PPPFXRPE6Q7X2QF3HHOT6Y6JSIACBRWUSNYJIBFRAGYIVPKEVHYA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.466197426 (588h43m26.337280471s) [noise=true , address=145.79.227.23:4545 , piece_id=5NVNPRC6MTKCYNXQAP4RVX66HKGAHFYZ3O4T6TBMMNXRPV5SCQSQ, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.466924340 (588h43m26.338007376s) [noise=true , address=145.79.227.35:4481 , piece_id=KCSQ4D4QXCRDBEVQ23NL2CUSVKINM6VEO5AJB7D64NJ5UOKR6KMA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.468159393 (588h43m26.339242428s) [noise=true , address=145.79.227.33:4871 , piece_id=VO6WL36M2PEQ4CH3TB5TSNQXMCHUW2CYSINHZ6UTP4OLUIZWXMHQ, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.471152222 (588h43m26.342235257s) [noise=true , address=145.79.227.10:4402 , piece_id=TZ2MT3M2VZAKKLRX7MX3RYBDKGASGMVJ6E3Z3L7TRR2MJGGKGWGQ, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:28.064015810 (588h43m27.935098855s) [upload=516094]: Next returned nil segment Dec 30 04:12:28.064028674 (588h43m27.935111709s) [upload=516094]: Waiting for error group managing segments... Dec 30 04:12:28.433157354 (588h43m28.304240389s) [noise=true , address=145.79.227.6:4258 , piece_id=VNQTOHC6AXNPTVVPOGJ5DYSGEUKRMFAY2GG3LJKOXYDEMZEKIIVQ, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.439536271 (588h43m28.310619316s) [noise=true , address=145.79.227.7:4294 , piece_id=VQXBFHDIMOA4AR2V5IU4D7WHGCTQOVQFS5FFVSMHIEL6RMXBUVSA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.446052540 (588h43m28.317135575s) [noise=true , address=145.79.227.8:4329 , piece_id=5PBTZTK3WCEYL7UNXDNK7ED5UZSBC7DQ56EKRFF3AKX6EM33C55A, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.449775599 (588h43m28.320858645s) [noise=true , address=145.79.227.41:4086 , piece_id=5ESIXVVZ5R6TZ7KC7OX7KURQ2BGHWQVLR7IOB6KHEUDAYXSOS7ZA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.450978080 (588h43m28.322061115s) [noise=true , address=145.79.227.9:4365 , piece_id=RC6CODBHZPOLTXYSSATCVYZGBHR6Q6BXPJVDOFRCQBRO4W2BGKGA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.454408401 (588h43m28.325491436s) [noise=true , address=145.79.227.23:4545 , piece_id=5NVNPRC6MTKCYNXQAP4RVX66HKGAHFYZ3O4T6TBMMNXRPV5SCQSQ, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.458105311 (588h43m28.329188346s) [noise=true , address=145.79.227.17:4971 , piece_id=2HZERGO7D2LCLXTTFVJRHL6REXQQR5FTIQX2QRZYWSR3BOOUEBFA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.461820194 (588h43m28.332903229s) [noise=true , address=145.79.227.10:4402 , piece_id=TZ2MT3M2VZAKKLRX7MX3RYBDKGASGMVJ6E3Z3L7TRR2MJGGKGWGQ, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.462558411 (588h43m28.333641446s) [noise=true , address=145.79.227.4:4185 , piece_id=VU2W4WHRCNFDB6DMQFJOOWA7RWLHMON36WCJEYBIOK76TIJDGY3A, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.473689927 (588h43m28.344772963s) [noise=true , address=145.79.227.42:4123 , piece_id=7UDGXR62LWGGWAKKSHEUCR5FGYEGZJMU3MW4AS3O7RPHCRPKS4OA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.724530027 (588h43m28.595613073s) [noise=true , address=45.91.116.9:4262 , piece_id=7IBPBSMR36JFMDXEIHHQHJGCNR44IQVRSK2LSIQIHK2ITYW2QTFA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.741946263 (588h43m28.613029308s) [noise=true , address=45.91.116.11:4120 , piece_id=PPPFXRPE6Q7X2QF3HHOT6Y6JSIACBRWUSNYJIBFRAGYIVPKEVHYA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:29.350263495 (588h43m29.22134653s) [noise=true , address=77.90.31.15:4700 , piece_id=HF672QJYI2LHHAIFL4KIH3KPYOJSAKSOKEMFAD3VRI6SDIFWWUUQ, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:29.350333468 (588h43m29.221416513s) [seg_pos={0 0}, upload=516094]: Segment reached optimal threshold of 13 pieces. Dec 30 04:12:29.350527718 (588h43m29.221610753s) [noise=true , address=145.79.227.35:4481 , piece_id=KCSQ4D4QXCRDBEVQ23NL2CUSVKINM6VEO5AJB7D64NJ5UOKR6KMA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 04:12:29.350537647 (588h43m29.221620682s) [noise=true , address=145.79.227.35:4481 , piece_id=KCSQ4D4QXCRDBEVQ23NL2CUSVKINM6VEO5AJB7D64NJ5UOKR6KMA, seg_pos={0 0}, upload=516094]: Long tail cancellation due to optimal threshold Dec 30 04:12:29.350689315 (588h43m29.22177236s) [noise=true , address=145.79.227.33:4871 , piece_id=VO6WL36M2PEQ4CH3TB5TSNQXMCHUW2CYSINHZ6UTP4OLUIZWXMHQ, seg_pos={0 0}, upload=516094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 30 04:12:29.350713472 (588h43m29.221796517s) [noise=true , address=145.79.227.33:4871 , piece_id=VO6WL36M2PEQ4CH3TB5TSNQXMCHUW2CYSINHZ6UTP4OLUIZWXMHQ, seg_pos={0 0}, upload=516094]: Long tail cancellation due to optimal threshold Dec 30 04:12:29.350849250 (588h43m29.221932295s) [noise=true , address=45.91.116.4:4940 , piece_id=F2QGTNWOE3E4XDFYPR6D5YLEYB7X74T6TGT4TUZ3XXN6WETBPIMA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 04:12:29.350863497 (588h43m29.221946552s) [noise=true , address=45.91.116.4:4940 , piece_id=F2QGTNWOE3E4XDFYPR6D5YLEYB7X74T6TGT4TUZ3XXN6WETBPIMA, seg_pos={0 0}, upload=516094]: Long tail cancellation due to optimal threshold Dec 30 04:12:29.350915526 (588h43m29.221998562s) [seg_pos={0 0}, upload=516094]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:12:29.350994016 (588h43m29.222077061s) [upload=516094]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:12:29.350999276 (588h43m29.222082311s) [upload=516094]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:12:29.638767606 (588h43m29.509850641s) [upload=516094]: Done uploading segments... Dec 30 04:12:29.638786313 (588h43m29.509869348s) [upload=516094]: Upload finished. err: <nil> Duration: 5.999155047s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X............X..............] duration: 47.763287966s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:12:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 47.763304408s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XXXX........XX.XX...XXXX...XXXX...X.X...XX..X.X..XX.......XX] duration: 30.004642659s trace: https://jaeger.dev.storj.io/trace/d7820782ab9ff69 FAIL (2025-12-30 04:12:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: 7IBPBSMR36JFMDXEIHHQHJGCNR44IQVRSK2LSIQIHK2ITYW2QTFA) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: HF672QJYI2LHHAIFL4KIH3KPYOJSAKSOKEMFAD3VRI6SDIFWWUUQ) context deadline exceeded ======LOGS====== Dec 30 04:12:26.206421831 (588h43m26.077504866s) [upload=516094]: Starting upload Dec 30 04:12:26.206542230 (588h43m26.077625275s) [upload=516094]: Done starting upload Dec 30 04:12:26.206613877 (588h43m26.077696922s) [upload=516094]: Uploading segments... Dec 30 04:12:26.206825821 (588h43m26.077908866s) [upload=516094]: Got next segment. Inline: false Dec 30 04:12:26.206892287 (588h43m26.077975333s) [upload=516094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:12:26.206899070 (588h43m26.077982115s) [upload=516094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:12:26.455180628 (588h43m26.326263663s) [seg_pos={0 0}, upload=516094]: Begin upload segment... Dec 30 04:12:26.455236104 (588h43m26.326319139s) [seg_pos={0 0}, upload=516094]: Done begin upload segment. Dec 30 04:12:26.455289436 (588h43m26.326372481s) [noise=true , address=145.79.227.4:4185 , piece_id=VU2W4WHRCNFDB6DMQFJOOWA7RWLHMON36WCJEYBIOK76TIJDGY3A, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.455306849 (588h43m26.326389884s) [noise=true , address=45.91.116.9:4262 , piece_id=7IBPBSMR36JFMDXEIHHQHJGCNR44IQVRSK2LSIQIHK2ITYW2QTFA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.455800058 (588h43m26.326883093s) [noise=true , address=145.79.227.8:4329 , piece_id=5PBTZTK3WCEYL7UNXDNK7ED5UZSBC7DQ56EKRFF3AKX6EM33C55A, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.456082475 (588h43m26.32716552s) [noise=true , address=45.91.116.4:4940 , piece_id=F2QGTNWOE3E4XDFYPR6D5YLEYB7X74T6TGT4TUZ3XXN6WETBPIMA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.456505972 (588h43m26.327589007s) [noise=true , address=145.79.227.7:4294 , piece_id=VQXBFHDIMOA4AR2V5IU4D7WHGCTQOVQFS5FFVSMHIEL6RMXBUVSA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.456939117 (588h43m26.328022162s) [noise=true , address=145.79.227.9:4365 , piece_id=RC6CODBHZPOLTXYSSATCVYZGBHR6Q6BXPJVDOFRCQBRO4W2BGKGA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.457286819 (588h43m26.328369854s) [noise=true , address=145.79.227.42:4123 , piece_id=7UDGXR62LWGGWAKKSHEUCR5FGYEGZJMU3MW4AS3O7RPHCRPKS4OA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.458782647 (588h43m26.329865692s) [noise=true , address=145.79.227.17:4971 , piece_id=2HZERGO7D2LCLXTTFVJRHL6REXQQR5FTIQX2QRZYWSR3BOOUEBFA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.460064408 (588h43m26.331147453s) [noise=true , address=145.79.227.41:4086 , piece_id=5ESIXVVZ5R6TZ7KC7OX7KURQ2BGHWQVLR7IOB6KHEUDAYXSOS7ZA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.461325209 (588h43m26.332408254s) [noise=true , address=77.90.31.15:4700 , piece_id=HF672QJYI2LHHAIFL4KIH3KPYOJSAKSOKEMFAD3VRI6SDIFWWUUQ, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.463047268 (588h43m26.334130313s) [noise=true , address=145.79.227.6:4258 , piece_id=VNQTOHC6AXNPTVVPOGJ5DYSGEUKRMFAY2GG3LJKOXYDEMZEKIIVQ, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.464893033 (588h43m26.335976068s) [noise=true , address=45.91.116.11:4120 , piece_id=PPPFXRPE6Q7X2QF3HHOT6Y6JSIACBRWUSNYJIBFRAGYIVPKEVHYA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.466197426 (588h43m26.337280471s) [noise=true , address=145.79.227.23:4545 , piece_id=5NVNPRC6MTKCYNXQAP4RVX66HKGAHFYZ3O4T6TBMMNXRPV5SCQSQ, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.466924340 (588h43m26.338007376s) [noise=true , address=145.79.227.35:4481 , piece_id=KCSQ4D4QXCRDBEVQ23NL2CUSVKINM6VEO5AJB7D64NJ5UOKR6KMA, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.468159393 (588h43m26.339242428s) [noise=true , address=145.79.227.33:4871 , piece_id=VO6WL36M2PEQ4CH3TB5TSNQXMCHUW2CYSINHZ6UTP4OLUIZWXMHQ, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:26.471152222 (588h43m26.342235257s) [noise=true , address=145.79.227.10:4402 , piece_id=TZ2MT3M2VZAKKLRX7MX3RYBDKGASGMVJ6E3Z3L7TRR2MJGGKGWGQ, seg_pos={0 0}, upload=516094]: Uploading piece... Dec 30 04:12:28.064015810 (588h43m27.935098855s) [upload=516094]: Next returned nil segment Dec 30 04:12:28.064028674 (588h43m27.935111709s) [upload=516094]: Waiting for error group managing segments... Dec 30 04:12:28.433157354 (588h43m28.304240389s) [noise=true , address=145.79.227.6:4258 , piece_id=VNQTOHC6AXNPTVVPOGJ5DYSGEUKRMFAY2GG3LJKOXYDEMZEKIIVQ, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.439536271 (588h43m28.310619316s) [noise=true , address=145.79.227.7:4294 , piece_id=VQXBFHDIMOA4AR2V5IU4D7WHGCTQOVQFS5FFVSMHIEL6RMXBUVSA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.446052540 (588h43m28.317135575s) [noise=true , address=145.79.227.8:4329 , piece_id=5PBTZTK3WCEYL7UNXDNK7ED5UZSBC7DQ56EKRFF3AKX6EM33C55A, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.449775599 (588h43m28.320858645s) [noise=true , address=145.79.227.41:4086 , piece_id=5ESIXVVZ5R6TZ7KC7OX7KURQ2BGHWQVLR7IOB6KHEUDAYXSOS7ZA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.450978080 (588h43m28.322061115s) [noise=true , address=145.79.227.9:4365 , piece_id=RC6CODBHZPOLTXYSSATCVYZGBHR6Q6BXPJVDOFRCQBRO4W2BGKGA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.454408401 (588h43m28.325491436s) [noise=true , address=145.79.227.23:4545 , piece_id=5NVNPRC6MTKCYNXQAP4RVX66HKGAHFYZ3O4T6TBMMNXRPV5SCQSQ, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.458105311 (588h43m28.329188346s) [noise=true , address=145.79.227.17:4971 , piece_id=2HZERGO7D2LCLXTTFVJRHL6REXQQR5FTIQX2QRZYWSR3BOOUEBFA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.461820194 (588h43m28.332903229s) [noise=true , address=145.79.227.10:4402 , piece_id=TZ2MT3M2VZAKKLRX7MX3RYBDKGASGMVJ6E3Z3L7TRR2MJGGKGWGQ, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.462558411 (588h43m28.333641446s) [noise=true , address=145.79.227.4:4185 , piece_id=VU2W4WHRCNFDB6DMQFJOOWA7RWLHMON36WCJEYBIOK76TIJDGY3A, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.473689927 (588h43m28.344772963s) [noise=true , address=145.79.227.42:4123 , piece_id=7UDGXR62LWGGWAKKSHEUCR5FGYEGZJMU3MW4AS3O7RPHCRPKS4OA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.724530027 (588h43m28.595613073s) [noise=true , address=45.91.116.9:4262 , piece_id=7IBPBSMR36JFMDXEIHHQHJGCNR44IQVRSK2LSIQIHK2ITYW2QTFA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:28.741946263 (588h43m28.613029308s) [noise=true , address=45.91.116.11:4120 , piece_id=PPPFXRPE6Q7X2QF3HHOT6Y6JSIACBRWUSNYJIBFRAGYIVPKEVHYA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:29.350263495 (588h43m29.22134653s) [noise=true , address=77.90.31.15:4700 , piece_id=HF672QJYI2LHHAIFL4KIH3KPYOJSAKSOKEMFAD3VRI6SDIFWWUUQ, seg_pos={0 0}, upload=516094]: Done uploading piece. err: <nil> Dec 30 04:12:29.350333468 (588h43m29.221416513s) [seg_pos={0 0}, upload=516094]: Segment reached optimal threshold of 13 pieces. Dec 30 04:12:29.350527718 (588h43m29.221610753s) [noise=true , address=145.79.227.35:4481 , piece_id=KCSQ4D4QXCRDBEVQ23NL2CUSVKINM6VEO5AJB7D64NJ5UOKR6KMA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 04:12:29.350537647 (588h43m29.221620682s) [noise=true , address=145.79.227.35:4481 , piece_id=KCSQ4D4QXCRDBEVQ23NL2CUSVKINM6VEO5AJB7D64NJ5UOKR6KMA, seg_pos={0 0}, upload=516094]: Long tail cancellation due to optimal threshold Dec 30 04:12:29.350689315 (588h43m29.22177236s) [noise=true , address=145.79.227.33:4871 , piece_id=VO6WL36M2PEQ4CH3TB5TSNQXMCHUW2CYSINHZ6UTP4OLUIZWXMHQ, seg_pos={0 0}, upload=516094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 30 04:12:29.350713472 (588h43m29.221796517s) [noise=true , address=145.79.227.33:4871 , piece_id=VO6WL36M2PEQ4CH3TB5TSNQXMCHUW2CYSINHZ6UTP4OLUIZWXMHQ, seg_pos={0 0}, upload=516094]: Long tail cancellation due to optimal threshold Dec 30 04:12:29.350849250 (588h43m29.221932295s) [noise=true , address=45.91.116.4:4940 , piece_id=F2QGTNWOE3E4XDFYPR6D5YLEYB7X74T6TGT4TUZ3XXN6WETBPIMA, seg_pos={0 0}, upload=516094]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 04:12:29.350863497 (588h43m29.221946552s) [noise=true , address=45.91.116.4:4940 , piece_id=F2QGTNWOE3E4XDFYPR6D5YLEYB7X74T6TGT4TUZ3XXN6WETBPIMA, seg_pos={0 0}, upload=516094]: Long tail cancellation due to optimal threshold Dec 30 04:12:29.350915526 (588h43m29.221998562s) [seg_pos={0 0}, upload=516094]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:12:29.350994016 (588h43m29.222077061s) [upload=516094]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:12:29.350999276 (588h43m29.222082311s) [upload=516094]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:12:29.638767606 (588h43m29.509850641s) [upload=516094]: Done uploading segments... Dec 30 04:12:29.638786313 (588h43m29.509869348s) [upload=516094]: Upload finished. err: <nil> Duration: 30.004657738s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.XX...X.......................X..XXX............XX..X.....X.] duration: 5.001519137s trace: https://jaeger.dev.storj.io/trace/408434392b868da5 FAIL (2025-12-30 04:12:36 UTC) orbiter warning: context deadline exceeded Duration: 5.001539957s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XXXX........XX.XX...XXXX...XXXX...X.X...XX..X.X..XX.......X] duration: 30.003453513s trace: https://jaeger.dev.storj.io/trace/7ec3fad788f456da FAIL (2025-12-30 04:12:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 04:11:55.970822957 (588h42m55.841905992s) [upload=516088]: Starting upload Dec 30 04:11:55.970914431 (588h42m55.841997506s) [upload=516088]: Done starting upload Dec 30 04:11:55.971055369 (588h42m55.842138404s) [upload=516088]: Uploading segments... Dec 30 04:11:55.971469448 (588h42m55.842552493s) [upload=516088]: Got next segment. Inline: false Dec 30 04:11:55.971545612 (588h42m55.842628647s) [upload=516088]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 04:11:55.971553578 (588h42m55.842636623s) [upload=516088]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 04:11:56.219375922 (588h42m56.090458957s) [seg_pos={0 0}, upload=516088]: Begin upload segment... Dec 30 04:11:56.219426708 (588h42m56.090509753s) [seg_pos={0 0}, upload=516088]: Done begin upload segment. Dec 30 04:11:56.219456084 (588h42m56.090539129s) [noise=true , address=45.91.116.8:4046 , piece_id=MWE5EUKGFGBYW46E4UVWQ2OOBGSLX2PTKLWP7DDXHBU3M5BH74TQ, seg_pos={0 0}, upload=516088]: Uploading piece... Dec 30 04:11:56.219484568 (588h42m56.090567614s) [noise=true , address=145.79.227.43:4159 , piece_id=QHZ2A23FQDFNLPJQWNP5ORYSIRTFO7S7YQZRLUSOJWGQPQS6B2UQ, seg_pos={0 0}, upload=516088]: Uploading piece... Dec 30 04:11:56.219990371 (588h42m56.091073416s) [noise=true , address=145.79.227.42:4122 , piece_id=YCSG6M23PN3KZJ6DTWLL4VHSBTSOKNZHOD4WGVKHZQ5XGTELVQQA, seg_pos={0 0}, upload=516088]: Uploading piece... Dec 30 04:11:56.220582028 (588h42m56.091665063s) [noise=true , address=145.79.227.52:4447 , piece_id=XTMXY6G665TWJ7RRTC4CMHRFWTZ3XFHFXXFTOQZ2KBQWZOJU44RA, seg_pos={0 0}, upload=516088]: Uploading piece... Dec 30 04:11:56.221057734 (588h42m56.092140769s) [noise=true , address=145.79.227.19:4077 , piece_id=C63TNQAANSTPXOZUU6GCO3RJTYR6YAZXOVBHLTRNJY55XXTEXCKQ, seg_pos={0 0}, upload=516088]: Uploading piece... Dec 30 04:11:56.221564169 (588h42m56.092647204s) [noise=true , address=145.79.227.2:4114 , piece_id=GKFBRHWFBVZQ4GDJK22VDE3AMOH4GXYNPQ7QUSHYJXIA36EKFF2Q, seg_pos={0 0}, upload=516088]: Uploading piece... Dec 30 04:11:56.222065713 (588h42m56.093148748s) [noise=true , address=145.79.227.21:4473 , piece_id=PPDXH6DX2ZRWDW2HMGG3AQSBPNAIHKVHFVBCWS6FSRY36PJTBZOQ, seg_pos={0 0}, upload=516088]: Uploading piece... Dec 30 04:11:56.223089212 (588h42m56.094172257s) [noise=true , address=145.79.227.45:4230 , piece_id=ZSPPLCSNTLD3I63A63RAKOYMBOQIV62DTF6E4TE2CU2LXNP5P6GA, seg_pos={0 0}, upload=516088]: Uploading piece... Dec 30 04:11:56.223533438 (588h42m56.094616483s) [noise=true , address=45.91.116.11:4120 , piece_id=FNJRBVNDWV3VHJTZNZ34CRNXLWDCDZQD5JHDPDOZQBUNCDIZXPRA, seg_pos={0 0}, upload=516088]: Uploading piece... Dec 30 04:11:56.224089377 (588h42m56.095172422s) [noise=true , address=145.79.227.37:4565 , piece_id=KF2PSRULVIHW3NGX3RU722KUCIATEBVYQGPNXHVS4SJAE2DQP5KQ, seg_pos={0 0}, upload=516088]: Uploading piece... Dec 30 04:11:56.224669852 (588h42m56.095752887s) [noise=true , address=145.79.227.47:4265 , piece_id=VZSM6M6K22BYEQ2YJ46GZPNF677LSZSEPTTYOTAZVLSQJOWDEYEQ, seg_pos={0 0}, upload=516088]: Uploading piece... Dec 30 04:11:56.225216784 (588h42m56.096299829s) [noise=true , address=77.90.31.21:4878 , piece_id=6COMEG3SGRTFBUKSHVJMHTJJJHJNZCE25ZUVWQDGYNFKA6AVMYQA, seg_pos={0 0}, upload=516088]: Uploading piece... Dec 30 04:11:56.225845721 (588h42m56.096928757s) [noise=true , address=145.79.227.6:4258 , piece_id=STBNPZKPT4AJJJH5KFPE6MUCPFGETYCKKPVFJWDKDFG3DZVAZWMA, seg_pos={0 0}, upload=516088]: Uploading piece... Dec 30 04:11:56.228068194 (588h42m56.099151239s) [noise=true , address=77.90.31.11:4555 , piece_id=WQWMCZLV4WQE4OBBQU4PJB2BVGUPLICEDNZ4ZYUXLB25BWHET5NQ, seg_pos={0 0}, upload=516088]: Uploading piece... Dec 30 04:11:56.230605124 (588h42m56.10168817s) [noise=true , address=77.90.31.10:4518 , piece_id=3UANJHRS7PJPSPQV236JL4GZCI6MEQWQDDO73DMKIDPM7M6BEO2A, seg_pos={0 0}, upload=516088]: Uploading piece... Dec 30 04:11:56.232203789 (588h42m56.103286844s) [noise=true , address=145.79.227.24:4582 , piece_id=SWXDHGZPFL2JUK3F67PAVKECWCLLX626HTSFH5P4EYJZ6ZK26LGQ, seg_pos={0 0}, upload=516088]: Uploading piece... Dec 30 04:11:57.918741836 (588h42m57.789824871s) [upload=516088]: Next returned nil segment Dec 30 04:11:57.918755081 (588h42m57.789838116s) [upload=516088]: Waiting for error group managing segments... Dec 30 04:11:58.198106003 (588h42m58.069189038s) [noise=true , address=145.79.227.45:4230 , piece_id=ZSPPLCSNTLD3I63A63RAKOYMBOQIV62DTF6E4TE2CU2LXNP5P6GA, seg_pos={0 0}, upload=516088]: Done uploading piece. err: <nil> Dec 30 04:11:58.201052644 (588h42m58.072135689s) [noise=true , address=145.79.227.2:4114 , piece_id=GKFBRHWFBVZQ4GDJK22VDE3AMOH4GXYNPQ7QUSHYJXIA36EKFF2Q, seg_pos={0 0}, upload=516088]: Done uploading piece. err: <nil> Dec 30 04:11:58.206586551 (588h42m58.077669597s) [noise=true , address=145.79.227.37:4565 , piece_id=KF2PSRULVIHW3NGX3RU722KUCIATEBVYQGPNXHVS4SJAE2DQP5KQ, seg_pos={0 0}, upload=516088]: Done uploading piece. err: <nil> Dec 30 04:11:58.212548314 (588h42m58.083631349s) [noise=true , address=145.79.227.6:4258 , piece_id=STBNPZKPT4AJJJH5KFPE6MUCPFGETYCKKPVFJWDKDFG3DZVAZWMA, seg_pos={0 0}, upload=516088]: Done uploading piece. err: <nil> Dec 30 04:11:58.213238709 (588h42m58.084321744s) [noise=true , address=145.79.227.21:4473 , piece_id=PPDXH6DX2ZRWDW2HMGG3AQSBPNAIHKVHFVBCWS6FSRY36PJTBZOQ, seg_pos={0 0}, upload=516088]: Done uploading piece. err: <nil> Dec 30 04:11:58.213737869 (588h42m58.084820914s) [noise=true , address=145.79.227.19:4077 , piece_id=C63TNQAANSTPXOZUU6GCO3RJTYR6YAZXOVBHLTRNJY55XXTEXCKQ, seg_pos={0 0}, upload=516088]: Done uploading piece. err: <nil> Dec 30 04:11:58.222427477 (588h42m58.093510522s) [noise=true , address=145.79.227.52:4447 , piece_id=XTMXY6G665TWJ7RRTC4CMHRFWTZ3XFHFXXFTOQZ2KBQWZOJU44RA, seg_pos={0 0}, upload=516088]: Done uploading piece. err: <nil> Dec 30 04:11:58.227394776 (588h42m58.098477821s) [noise=true , address=145.79.227.24:4582 , piece_id=SWXDHGZPFL2JUK3F67PAVKECWCLLX626HTSFH5P4EYJZ6ZK26LGQ, seg_pos={0 0}, upload=516088]: Done uploading piece. err: <nil> Dec 30 04:11:58.233032932 (588h42m58.104115967s) [noise=true , address=145.79.227.42:4122 , piece_id=YCSG6M23PN3KZJ6DTWLL4VHSBTSOKNZHOD4WGVKHZQ5XGTELVQQA, seg_pos={0 0}, upload=516088]: Done uploading piece. err: <nil> Dec 30 04:11:58.243952836 (588h42m58.115035881s) [noise=true , address=145.79.227.47:4265 , piece_id=VZSM6M6K22BYEQ2YJ46GZPNF677LSZSEPTTYOTAZVLSQJOWDEYEQ, seg_pos={0 0}, upload=516088]: Done uploading piece. err: <nil> Dec 30 04:11:58.430895480 (588h42m58.301978515s) [noise=true , address=77.90.31.21:4878 , piece_id=6COMEG3SGRTFBUKSHVJMHTJJJHJNZCE25ZUVWQDGYNFKA6AVMYQA, seg_pos={0 0}, upload=516088]: Done uploading piece. err: <nil> Dec 30 04:11:58.528961561 (588h42m58.400044606s) [noise=true , address=77.90.31.11:4555 , piece_id=WQWMCZLV4WQE4OBBQU4PJB2BVGUPLICEDNZ4ZYUXLB25BWHET5NQ, seg_pos={0 0}, upload=516088]: Done uploading piece. err: <nil> Dec 30 04:11:58.784472977 (588h42m58.655556022s) [noise=true , address=45.91.116.8:4046 , piece_id=MWE5EUKGFGBYW46E4UVWQ2OOBGSLX2PTKLWP7DDXHBU3M5BH74TQ, seg_pos={0 0}, upload=516088]: Done uploading piece. err: <nil> Dec 30 04:11:58.784510338 (588h42m58.655593373s) [seg_pos={0 0}, upload=516088]: Segment reached optimal threshold of 13 pieces. Dec 30 04:11:58.784805690 (588h42m58.655888735s) [noise=true , address=45.91.116.11:4120 , piece_id=FNJRBVNDWV3VHJTZNZ34CRNXLWDCDZQD5JHDPDOZQBUNCDIZXPRA, seg_pos={0 0}, upload=516088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 188.166.225.170:42384->45.91.116.11:4120: use of closed network connection; write tcp 188.166.225.170:42384->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42384->45.91.116.11:4120: use of closed network connection Dec 30 04:11:58.784813936 (588h42m58.655896981s) [noise=true , address=45.91.116.11:4120 , piece_id=FNJRBVNDWV3VHJTZNZ34CRNXLWDCDZQD5JHDPDOZQBUNCDIZXPRA, seg_pos={0 0}, upload=516088]: Long tail cancellation due to optimal threshold Dec 30 04:11:58.785158392 (588h42m58.656241427s) [noise=true , address=77.90.31.10:4518 , piece_id=3UANJHRS7PJPSPQV236JL4GZCI6MEQWQDDO73DMKIDPM7M6BEO2A, seg_pos={0 0}, upload=516088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 30 04:11:58.785167119 (588h42m58.656250154s) [noise=true , address=77.90.31.10:4518 , piece_id=3UANJHRS7PJPSPQV236JL4GZCI6MEQWQDDO73DMKIDPM7M6BEO2A, seg_pos={0 0}, upload=516088]: Long tail cancellation due to optimal threshold Dec 30 04:11:58.785296855 (588h42m58.6563799s) [noise=true , address=145.79.227.43:4159 , piece_id=QHZ2A23FQDFNLPJQWNP5ORYSIRTFO7S7YQZRLUSOJWGQPQS6B2UQ, seg_pos={0 0}, upload=516088]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 30 04:11:58.785304651 (588h42m58.656387686s) [noise=true , address=145.79.227.43:4159 , piece_id=QHZ2A23FQDFNLPJQWNP5ORYSIRTFO7S7YQZRLUSOJWGQPQS6B2UQ, seg_pos={0 0}, upload=516088]: Long tail cancellation due to optimal threshold Dec 30 04:11:58.785326812 (588h42m58.656409847s) [seg_pos={0 0}, upload=516088]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 04:11:58.785403178 (588h42m58.656486223s) [upload=516088]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:11:58.785409010 (588h42m58.656492045s) [upload=516088]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:11:59.070127933 (588h42m58.941210968s) [upload=516088]: Done uploading segments... Dec 30 04:11:59.070203627 (588h42m58.941286662s) [upload=516088]: Upload finished. err: <nil> Duration: 30.003474233s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X............X.............] duration: 22.176656973s trace: https://jaeger.dev.storj.io/trace/637c536fc008fb78 FAIL (2025-12-30 04:12:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 2m8.645593264s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.775911497 (588h32m8.646994542s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.777318166 (588h32m8.648401212s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778059658 (588h32m8.649142703s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778338178 (588h32m8.649421224s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.778723793 (588h32m8.649806838s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.779305430 (588h32m8.650388476s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780090406 (588h32m8.651173441s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.780617810 (588h32m8.651700855s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.781367337 (588h32m8.652450373s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782194792 (588h32m8.653277837s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.782953948 (588h32m8.654036993s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.786503808 (588h32m8.657586853s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.788837112 (588h32m8.659920157s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.793855728 (588h32m8.664938773s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:08.795240425 (588h32m8.66632346s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Uploading piece... Dec 30 04:01:10.180178281 (588h32m10.051261327s) [upload=515927]: Next returned nil segment Dec 30 04:01:10.180193331 (588h32m10.051276366s) [upload=515927]: Waiting for error group managing segments... Dec 30 04:01:10.630495634 (588h32m10.501578669s) [noise=true , address=145.79.227.2:4125 , piece_id=BLLI3M637UHJRZP3YL52RE62RUEVGLHZXVAXPBZVJZMQPW5ZTIUA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.659778345 (588h32m10.53086139s) [noise=true , address=145.79.227.43:4160 , piece_id=4LCCWMEXVCUPMLLKG26S5EMIJP3MDEDNZNVLZRGJJOJJE7Q7AADA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.671799526 (588h32m10.542882562s) [noise=true , address=145.79.227.19:4083 , piece_id=MCQJI74FZW4KRT3ZLL2W226HF5LAZFUU5ING327XISXQYV3CQ5WQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.676331005 (588h32m10.54741405s) [noise=true , address=145.79.227.40:4076 , piece_id=F63EK3TCIQFHVCCDV5DL5I2QTZJZYR7DRPOMFS6J3UEC5W6YX6BQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.820832219 (588h32m10.691915264s) [noise=true , address=77.90.31.15:4731 , piece_id=K3K6CN5V7SETRPMRCKHAVCNNK4DPSYJCHSAJ6HEFWVQS5TCWJZAQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:10.843204321 (588h32m10.714287356s) [noise=true , address=145.79.227.35:4501 , piece_id=CAPYASPEUC4KMTXKL7V5VWAYCRY5752BPH5RKWUZVVU3HCVIAVLQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.434118945 (588h32m11.30520199s) [noise=true , address=193.149.164.143:4701 , piece_id=3QVRO2CR2P3YINLVIEUJTC3OBOYLKHIDVVWX5DSSGIWVBMK6WAKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.518819916 (588h32m11.389902961s) [noise=true , address=77.90.31.19:4856 , piece_id=YJLYJIBVIXIYKZ33TVLDL4MPCDIEN3OX7FETKLSGSZAD3EUY4BPQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.573041516 (588h32m11.444124551s) [noise=true , address=145.79.227.44:4209 , piece_id=CBBJPUQKEQ6XT2CLE24WY4XISA2KKOGBTDZXHRTUO4TVD2ALQKXA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.584552196 (588h32m11.455635241s) [noise=true , address=45.91.116.12:4253 , piece_id=5TOZUTOACBYBHCH62TBWINFAN6GF466FU2JYJQJMGIPM7Y5HML7Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.588999815 (588h32m11.46008285s) [noise=true , address=64.49.13.154:4926 , piece_id=T5REBW4C4OAX3NHQIQ4JNMAJHLHSL77P43OAGOAAVEQDELYSXOJA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.595442002 (588h32m11.466525037s) [noise=true , address=145.79.227.34:4933 , piece_id=FLSB6H6OYEN4TZQFPP6DJBVKR2D6EUPQRTNAN52E6H3OTCZWXBBA, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.601502722 (588h32m11.472585767s) [noise=true , address=145.79.227.49:4358 , piece_id=TVR7OXHFNJN6J7J4HXPMHEC53NUX4NI74L6VGYITLHAQTSK54QIQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603589166 (588h32m11.474672201s) [noise=true , address=145.79.227.30:4795 , piece_id=ECOXYC77MJSY3VWOOPNY3YSX7H723EW2CKQIIYIPQCKCSVD3FANQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: <nil> Dec 30 04:01:11.603605928 (588h32m11.474688963s) [seg_pos={0 0}, upload=515927]: Segment reached optimal threshold of 14 pieces. Dec 30 04:01:11.603761845 (588h32m11.47484489s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.603767786 (588h32m11.474850821s) [noise=true , address=145.79.227.6:4285 , piece_id=P7OO3JQHCCYYN26PYMUGX2DROPI63B5BELMTI22GGARK2E7L4C5Q, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604070503 (588h32m11.475153538s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KrMw5DoiMSiBHMZJ8Zf8ir7UnTzBwUt9XMhw3Ria6ASGkm3FA): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604079330 (588h32m11.475162375s) [noise=true , address=77.90.31.13:4652 , piece_id=TNSSJ3SDTQUXNQ2DVRGRSVTEEYC3JWHVOS4JOTLJBKGM6522JSKQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604224596 (588h32m11.475307631s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14): protocol: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43318->145.79.227.5:4253: use of closed network connection Dec 30 04:01:11.604233653 (588h32m11.475316698s) [noise=true , address=145.79.227.5:4253 , piece_id=TA4BHMWSSF55XGHUNRG6K2L5XWKITIZMUP62ZRZGWD46UWJP54IQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604355586 (588h32m11.475438621s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga): protocol: expected piece hash; context canceled; EOF Dec 30 04:01:11.604361457 (588h32m11.475444502s) [noise=true , address=77.90.31.21:4908 , piece_id=WWHAPV3K3YCORFDS4GHD4U6C34PN25YUMDGMRIFBO4QVERGMH7GQ, seg_pos={0 0}, upload=515927]: Long tail cancellation due to optimal threshold Dec 30 04:01:11.604390893 (588h32m11.475473928s) [seg_pos={0 0}, upload=515927]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 04:01:11.604455616 (588h32m11.475538661s) [upload=515927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 04:01:11.604461698 (588h32m11.475544733s) [upload=515927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 04:01:11.911203579 (588h32m11.782286634s) [upload=515927]: Done uploading segments... Dec 30 04:01:11.911228716 (588h32m11.782311761s) [upload=515927]: Upload finished. err: <nil> Duration: 22.176688793s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [..XX...X.......................X..XXX............XX..X.....X] duration: 5.0007574s trace: https://jaeger.dev.storj.io/trace/408434392b868da5 FAIL (2025-12-30 04:11:36 UTC) orbiter warning: context deadline exceeded Duration: 5.000779351s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)