Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........X..........................................X.....] duration: 3.49656633s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:39:21 UTC) orbiter warning: context deadline exceeded Duration: 3.496628297s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............X..........................................X....] duration: 3.59762984s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:38:58 UTC) orbiter warning: context deadline exceeded Duration: 3.597652683s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...X..........................XXX.....XX.XX.XXXX.XXXXXXX.X..] duration: 16.220365211s trace: https://jaeger.dev.storj.io/trace/1b0d84de511f50ee FAIL (2025-12-27 23:38:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 23:37:09.461918156 (536h8m9.333001201s) [upload=470202]: Uploading segments... Dec 27 23:37:09.462109541 (536h8m9.333192586s) [upload=470202]: Got next segment. Inline: false Dec 27 23:37:09.462152402 (536h8m9.333235447s) [upload=470202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:37:09.462158183 (536h8m9.333241218s) [upload=470202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:37:09.706418395 (536h8m9.57750144s) [seg_pos={0 0}, upload=470202]: Begin upload segment... Dec 27 23:37:09.706493988 (536h8m9.577577033s) [seg_pos={0 0}, upload=470202]: Done begin upload segment. Dec 27 23:37:09.706530578 (536h8m9.577613613s) [noise=true , address=145.79.227.4:4187 , piece_id=NE6544KWVOUNMKMSARMSJELWHKYGWFHMOFANGJ7MSWNV3EEO4JMA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.707863706 (536h8m9.578946741s) [noise=true , address=45.91.116.11:4118 , piece_id=UDYUJETC735R5WWPE57TRP54BHLL7DTYCYPUNJGZ7RFPUHXF4WYQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.708961978 (536h8m9.580045013s) [noise=true , address=145.79.227.15:4874 , piece_id=N24MZCXSRSPNHTEJSSNTMAW6DU3EUXQPQOP2P5LAEQNMJS6GQREQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.709447032 (536h8m9.580530077s) [noise=true , address=77.90.31.11:4553 , piece_id=WRVJKWKRWIOE3IZ2WW3G5IUGZSA3XBFBTB2UUEDUNBZKI2KDTBQA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.709959948 (536h8m9.581042993s) [noise=true , address=77.90.31.19:4844 , piece_id=3YYYDVPMQSMCHVR2KZONCYIAGEBYFQMOECJQDKEY4XUYOIXZJCCQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.710579397 (536h8m9.581662452s) [noise=true , address=145.79.227.7:4293 , piece_id=M7YGTUL6NE2OLIZYSMF4KDE27HIIYBWIZEOQFUK3Q34HZCVY42UA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.711623495 (536h8m9.582706541s) [noise=true , address=145.79.227.50:4375 , piece_id=7Q5LXWSZCBNCFT6MZLWC2Y5ZYTZ4AGUS5YDCUMWZKGPGRBUYGODA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.713574480 (536h8m9.584657515s) [noise=true , address=77.90.31.16:4735 , piece_id=YWX7IWYYWS5KHGIXB2GUDWLLR4VKKHNIWMSEM3NPVEA6XPDEFHIQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.714036039 (536h8m9.585119084s) [noise=true , address=145.79.227.3:4150 , piece_id=ZOHJ4EEYJXBHHGXGR2ZRFZU2XVW54RE4ZZK7VY5HPJJY5YQL4SEQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.715228480 (536h8m9.586311515s) [noise=true , address=45.91.116.5:4299 , piece_id=653BZ7QJ4QS2FASQIUHLZMIFSEJ7Q54JQ4LHOAZRMQRCBROMORDA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.716142671 (536h8m9.587225716s) [noise=true , address=45.91.116.8:4048 , piece_id=Y5ERSOIELOPSTMS7KAGXATGFLFDU5MHCVYUXV6DB3LO5XLIA4BLQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.716790665 (536h8m9.5878737s) [noise=true , address=145.79.227.5:4223 , piece_id=SB2LII7QIPLOPWBHIHMWUE4X3FKW5C3UZZ23TL2UA7KMXTPR6JIQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.718649134 (536h8m9.589732169s) [noise=true , address=145.79.227.32:4836 , piece_id=A7WCN4AN645HPU3O3U5GX57FFVJKGTWNYIP3IRITJXFJV2WSVIVA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.720527691 (536h8m9.591610736s) [noise=true , address=145.79.227.52:4447 , piece_id=NMH7HHAI36XMOOT4SWLRFP22V4PMFODYI6XC5TQRQFNVFJGKB4IQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.724107828 (536h8m9.595190863s) [noise=true , address=145.79.227.11:4730 , piece_id=MYVD2VI5SSWEHKFGL246TYSOF6GWTKV7KPDUHDTTKYDAGPFUFVXA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.729567955 (536h8m9.60065099s) [noise=true , address=145.79.227.36:4517 , piece_id=XT3WVKQBKI326NGHAPF6UI3DKYIHHROQM6F52LIWWOUEKCCAXCMA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:11.436224761 (536h8m11.307307816s) [upload=470202]: Next returned nil segment Dec 27 23:37:11.436238377 (536h8m11.307321412s) [upload=470202]: Waiting for error group managing segments... Dec 27 23:37:11.711936292 (536h8m11.583019327s) [noise=true , address=145.79.227.15:4874 , piece_id=N24MZCXSRSPNHTEJSSNTMAW6DU3EUXQPQOP2P5LAEQNMJS6GQREQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.723417453 (536h8m11.594500498s) [noise=true , address=145.79.227.32:4836 , piece_id=A7WCN4AN645HPU3O3U5GX57FFVJKGTWNYIP3IRITJXFJV2WSVIVA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.727389777 (536h8m11.598472822s) [noise=true , address=145.79.227.11:4730 , piece_id=MYVD2VI5SSWEHKFGL246TYSOF6GWTKV7KPDUHDTTKYDAGPFUFVXA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.730891145 (536h8m11.60197418s) [noise=true , address=145.79.227.4:4187 , piece_id=NE6544KWVOUNMKMSARMSJELWHKYGWFHMOFANGJ7MSWNV3EEO4JMA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.734532098 (536h8m11.605615133s) [noise=true , address=145.79.227.52:4447 , piece_id=NMH7HHAI36XMOOT4SWLRFP22V4PMFODYI6XC5TQRQFNVFJGKB4IQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.735505420 (536h8m11.606588456s) [noise=true , address=145.79.227.7:4293 , piece_id=M7YGTUL6NE2OLIZYSMF4KDE27HIIYBWIZEOQFUK3Q34HZCVY42UA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.735992227 (536h8m11.607075273s) [noise=true , address=145.79.227.36:4517 , piece_id=XT3WVKQBKI326NGHAPF6UI3DKYIHHROQM6F52LIWWOUEKCCAXCMA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.739683547 (536h8m11.610766582s) [noise=true , address=145.79.227.3:4150 , piece_id=ZOHJ4EEYJXBHHGXGR2ZRFZU2XVW54RE4ZZK7VY5HPJJY5YQL4SEQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.746473637 (536h8m11.617556682s) [noise=true , address=145.79.227.5:4223 , piece_id=SB2LII7QIPLOPWBHIHMWUE4X3FKW5C3UZZ23TL2UA7KMXTPR6JIQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.748267662 (536h8m11.619350697s) [noise=true , address=145.79.227.50:4375 , piece_id=7Q5LXWSZCBNCFT6MZLWC2Y5ZYTZ4AGUS5YDCUMWZKGPGRBUYGODA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.909048586 (536h8m11.780131621s) [noise=true , address=77.90.31.19:4844 , piece_id=3YYYDVPMQSMCHVR2KZONCYIAGEBYFQMOECJQDKEY4XUYOIXZJCCQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.969787924 (536h8m11.840870959s) [noise=true , address=45.91.116.11:4118 , piece_id=UDYUJETC735R5WWPE57TRP54BHLL7DTYCYPUNJGZ7RFPUHXF4WYQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.981975691 (536h8m11.853058726s) [noise=true , address=77.90.31.16:4735 , piece_id=YWX7IWYYWS5KHGIXB2GUDWLLR4VKKHNIWMSEM3NPVEA6XPDEFHIQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.981992322 (536h8m11.853075357s) [seg_pos={0 0}, upload=470202]: Segment reached optimal threshold of 13 pieces. Dec 27 23:37:11.982300379 (536h8m11.853383424s) [noise=true , address=45.91.116.8:4048 , piece_id=Y5ERSOIELOPSTMS7KAGXATGFLFDU5MHCVYUXV6DB3LO5XLIA4BLQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:36568->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:36568->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36568->45.91.116.8:4048: use of closed network connection Dec 27 23:37:11.982310097 (536h8m11.853393142s) [noise=true , address=45.91.116.8:4048 , piece_id=Y5ERSOIELOPSTMS7KAGXATGFLFDU5MHCVYUXV6DB3LO5XLIA4BLQ, seg_pos={0 0}, upload=470202]: Long tail cancellation due to optimal threshold Dec 27 23:37:11.982579100 (536h8m11.853662145s) [noise=true , address=45.91.116.5:4299 , piece_id=653BZ7QJ4QS2FASQIUHLZMIFSEJ7Q54JQ4LHOAZRMQRCBROMORDA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 27 23:37:11.982587776 (536h8m11.853670811s) [noise=true , address=45.91.116.5:4299 , piece_id=653BZ7QJ4QS2FASQIUHLZMIFSEJ7Q54JQ4LHOAZRMQRCBROMORDA, seg_pos={0 0}, upload=470202]: Long tail cancellation due to optimal threshold Dec 27 23:37:11.982662508 (536h8m11.853745553s) [noise=true , address=77.90.31.11:4553 , piece_id=WRVJKWKRWIOE3IZ2WW3G5IUGZSA3XBFBTB2UUEDUNBZKI2KDTBQA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:33516->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:33516->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33516->77.90.31.11:4553: use of closed network connection Dec 27 23:37:11.982673981 (536h8m11.853757026s) [noise=true , address=77.90.31.11:4553 , piece_id=WRVJKWKRWIOE3IZ2WW3G5IUGZSA3XBFBTB2UUEDUNBZKI2KDTBQA, seg_pos={0 0}, upload=470202]: Long tail cancellation due to optimal threshold Dec 27 23:37:11.982696844 (536h8m11.853779879s) [seg_pos={0 0}, upload=470202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:37:11.982788629 (536h8m11.853871664s) [upload=470202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:37:11.982795271 (536h8m11.853878306s) [upload=470202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:37:12.273683530 (536h8m12.144766565s) [upload=470202]: Done uploading segments... Dec 27 23:37:12.273704841 (536h8m12.144787876s) [upload=470202]: Upload finished. err: <nil> Duration: 16.220396651s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [....X..........................XXX.....XX.XX.XXXX.XXXXXXX.X.] duration: 11.98213426s trace: https://jaeger.dev.storj.io/trace/1b0d84de511f50ee FAIL (2025-12-27 23:38:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 23:37:09.461918156 (536h8m9.333001201s) [upload=470202]: Uploading segments... Dec 27 23:37:09.462109541 (536h8m9.333192586s) [upload=470202]: Got next segment. Inline: false Dec 27 23:37:09.462152402 (536h8m9.333235447s) [upload=470202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:37:09.462158183 (536h8m9.333241218s) [upload=470202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:37:09.706418395 (536h8m9.57750144s) [seg_pos={0 0}, upload=470202]: Begin upload segment... Dec 27 23:37:09.706493988 (536h8m9.577577033s) [seg_pos={0 0}, upload=470202]: Done begin upload segment. Dec 27 23:37:09.706530578 (536h8m9.577613613s) [noise=true , address=145.79.227.4:4187 , piece_id=NE6544KWVOUNMKMSARMSJELWHKYGWFHMOFANGJ7MSWNV3EEO4JMA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.707863706 (536h8m9.578946741s) [noise=true , address=45.91.116.11:4118 , piece_id=UDYUJETC735R5WWPE57TRP54BHLL7DTYCYPUNJGZ7RFPUHXF4WYQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.708961978 (536h8m9.580045013s) [noise=true , address=145.79.227.15:4874 , piece_id=N24MZCXSRSPNHTEJSSNTMAW6DU3EUXQPQOP2P5LAEQNMJS6GQREQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.709447032 (536h8m9.580530077s) [noise=true , address=77.90.31.11:4553 , piece_id=WRVJKWKRWIOE3IZ2WW3G5IUGZSA3XBFBTB2UUEDUNBZKI2KDTBQA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.709959948 (536h8m9.581042993s) [noise=true , address=77.90.31.19:4844 , piece_id=3YYYDVPMQSMCHVR2KZONCYIAGEBYFQMOECJQDKEY4XUYOIXZJCCQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.710579397 (536h8m9.581662452s) [noise=true , address=145.79.227.7:4293 , piece_id=M7YGTUL6NE2OLIZYSMF4KDE27HIIYBWIZEOQFUK3Q34HZCVY42UA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.711623495 (536h8m9.582706541s) [noise=true , address=145.79.227.50:4375 , piece_id=7Q5LXWSZCBNCFT6MZLWC2Y5ZYTZ4AGUS5YDCUMWZKGPGRBUYGODA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.713574480 (536h8m9.584657515s) [noise=true , address=77.90.31.16:4735 , piece_id=YWX7IWYYWS5KHGIXB2GUDWLLR4VKKHNIWMSEM3NPVEA6XPDEFHIQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.714036039 (536h8m9.585119084s) [noise=true , address=145.79.227.3:4150 , piece_id=ZOHJ4EEYJXBHHGXGR2ZRFZU2XVW54RE4ZZK7VY5HPJJY5YQL4SEQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.715228480 (536h8m9.586311515s) [noise=true , address=45.91.116.5:4299 , piece_id=653BZ7QJ4QS2FASQIUHLZMIFSEJ7Q54JQ4LHOAZRMQRCBROMORDA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.716142671 (536h8m9.587225716s) [noise=true , address=45.91.116.8:4048 , piece_id=Y5ERSOIELOPSTMS7KAGXATGFLFDU5MHCVYUXV6DB3LO5XLIA4BLQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.716790665 (536h8m9.5878737s) [noise=true , address=145.79.227.5:4223 , piece_id=SB2LII7QIPLOPWBHIHMWUE4X3FKW5C3UZZ23TL2UA7KMXTPR6JIQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.718649134 (536h8m9.589732169s) [noise=true , address=145.79.227.32:4836 , piece_id=A7WCN4AN645HPU3O3U5GX57FFVJKGTWNYIP3IRITJXFJV2WSVIVA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.720527691 (536h8m9.591610736s) [noise=true , address=145.79.227.52:4447 , piece_id=NMH7HHAI36XMOOT4SWLRFP22V4PMFODYI6XC5TQRQFNVFJGKB4IQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.724107828 (536h8m9.595190863s) [noise=true , address=145.79.227.11:4730 , piece_id=MYVD2VI5SSWEHKFGL246TYSOF6GWTKV7KPDUHDTTKYDAGPFUFVXA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.729567955 (536h8m9.60065099s) [noise=true , address=145.79.227.36:4517 , piece_id=XT3WVKQBKI326NGHAPF6UI3DKYIHHROQM6F52LIWWOUEKCCAXCMA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:11.436224761 (536h8m11.307307816s) [upload=470202]: Next returned nil segment Dec 27 23:37:11.436238377 (536h8m11.307321412s) [upload=470202]: Waiting for error group managing segments... Dec 27 23:37:11.711936292 (536h8m11.583019327s) [noise=true , address=145.79.227.15:4874 , piece_id=N24MZCXSRSPNHTEJSSNTMAW6DU3EUXQPQOP2P5LAEQNMJS6GQREQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.723417453 (536h8m11.594500498s) [noise=true , address=145.79.227.32:4836 , piece_id=A7WCN4AN645HPU3O3U5GX57FFVJKGTWNYIP3IRITJXFJV2WSVIVA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.727389777 (536h8m11.598472822s) [noise=true , address=145.79.227.11:4730 , piece_id=MYVD2VI5SSWEHKFGL246TYSOF6GWTKV7KPDUHDTTKYDAGPFUFVXA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.730891145 (536h8m11.60197418s) [noise=true , address=145.79.227.4:4187 , piece_id=NE6544KWVOUNMKMSARMSJELWHKYGWFHMOFANGJ7MSWNV3EEO4JMA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.734532098 (536h8m11.605615133s) [noise=true , address=145.79.227.52:4447 , piece_id=NMH7HHAI36XMOOT4SWLRFP22V4PMFODYI6XC5TQRQFNVFJGKB4IQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.735505420 (536h8m11.606588456s) [noise=true , address=145.79.227.7:4293 , piece_id=M7YGTUL6NE2OLIZYSMF4KDE27HIIYBWIZEOQFUK3Q34HZCVY42UA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.735992227 (536h8m11.607075273s) [noise=true , address=145.79.227.36:4517 , piece_id=XT3WVKQBKI326NGHAPF6UI3DKYIHHROQM6F52LIWWOUEKCCAXCMA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.739683547 (536h8m11.610766582s) [noise=true , address=145.79.227.3:4150 , piece_id=ZOHJ4EEYJXBHHGXGR2ZRFZU2XVW54RE4ZZK7VY5HPJJY5YQL4SEQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.746473637 (536h8m11.617556682s) [noise=true , address=145.79.227.5:4223 , piece_id=SB2LII7QIPLOPWBHIHMWUE4X3FKW5C3UZZ23TL2UA7KMXTPR6JIQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.748267662 (536h8m11.619350697s) [noise=true , address=145.79.227.50:4375 , piece_id=7Q5LXWSZCBNCFT6MZLWC2Y5ZYTZ4AGUS5YDCUMWZKGPGRBUYGODA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.909048586 (536h8m11.780131621s) [noise=true , address=77.90.31.19:4844 , piece_id=3YYYDVPMQSMCHVR2KZONCYIAGEBYFQMOECJQDKEY4XUYOIXZJCCQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.969787924 (536h8m11.840870959s) [noise=true , address=45.91.116.11:4118 , piece_id=UDYUJETC735R5WWPE57TRP54BHLL7DTYCYPUNJGZ7RFPUHXF4WYQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.981975691 (536h8m11.853058726s) [noise=true , address=77.90.31.16:4735 , piece_id=YWX7IWYYWS5KHGIXB2GUDWLLR4VKKHNIWMSEM3NPVEA6XPDEFHIQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.981992322 (536h8m11.853075357s) [seg_pos={0 0}, upload=470202]: Segment reached optimal threshold of 13 pieces. Dec 27 23:37:11.982300379 (536h8m11.853383424s) [noise=true , address=45.91.116.8:4048 , piece_id=Y5ERSOIELOPSTMS7KAGXATGFLFDU5MHCVYUXV6DB3LO5XLIA4BLQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:36568->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:36568->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36568->45.91.116.8:4048: use of closed network connection Dec 27 23:37:11.982310097 (536h8m11.853393142s) [noise=true , address=45.91.116.8:4048 , piece_id=Y5ERSOIELOPSTMS7KAGXATGFLFDU5MHCVYUXV6DB3LO5XLIA4BLQ, seg_pos={0 0}, upload=470202]: Long tail cancellation due to optimal threshold Dec 27 23:37:11.982579100 (536h8m11.853662145s) [noise=true , address=45.91.116.5:4299 , piece_id=653BZ7QJ4QS2FASQIUHLZMIFSEJ7Q54JQ4LHOAZRMQRCBROMORDA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 27 23:37:11.982587776 (536h8m11.853670811s) [noise=true , address=45.91.116.5:4299 , piece_id=653BZ7QJ4QS2FASQIUHLZMIFSEJ7Q54JQ4LHOAZRMQRCBROMORDA, seg_pos={0 0}, upload=470202]: Long tail cancellation due to optimal threshold Dec 27 23:37:11.982662508 (536h8m11.853745553s) [noise=true , address=77.90.31.11:4553 , piece_id=WRVJKWKRWIOE3IZ2WW3G5IUGZSA3XBFBTB2UUEDUNBZKI2KDTBQA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:33516->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:33516->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33516->77.90.31.11:4553: use of closed network connection Dec 27 23:37:11.982673981 (536h8m11.853757026s) [noise=true , address=77.90.31.11:4553 , piece_id=WRVJKWKRWIOE3IZ2WW3G5IUGZSA3XBFBTB2UUEDUNBZKI2KDTBQA, seg_pos={0 0}, upload=470202]: Long tail cancellation due to optimal threshold Dec 27 23:37:11.982696844 (536h8m11.853779879s) [seg_pos={0 0}, upload=470202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:37:11.982788629 (536h8m11.853871664s) [upload=470202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:37:11.982795271 (536h8m11.853878306s) [upload=470202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:37:12.273683530 (536h8m12.144766565s) [upload=470202]: Done uploading segments... Dec 27 23:37:12.273704841 (536h8m12.144787876s) [upload=470202]: Upload finished. err: <nil> Duration: 11.982295166s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............X..........................................X...] duration: 3.466697304s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:38:18 UTC) orbiter warning: context deadline exceeded Duration: 3.466720758s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.....X..........................XXX.....XX.XX.XXXX.XXXXXXX.X] duration: 30.002892914s trace: https://jaeger.dev.storj.io/trace/1b0d84de511f50ee FAIL (2025-12-27 23:37:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 23:37:09.461918156 (536h8m9.333001201s) [upload=470202]: Uploading segments... Dec 27 23:37:09.462109541 (536h8m9.333192586s) [upload=470202]: Got next segment. Inline: false Dec 27 23:37:09.462152402 (536h8m9.333235447s) [upload=470202]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:37:09.462158183 (536h8m9.333241218s) [upload=470202]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:37:09.706418395 (536h8m9.57750144s) [seg_pos={0 0}, upload=470202]: Begin upload segment... Dec 27 23:37:09.706493988 (536h8m9.577577033s) [seg_pos={0 0}, upload=470202]: Done begin upload segment. Dec 27 23:37:09.706530578 (536h8m9.577613613s) [noise=true , address=145.79.227.4:4187 , piece_id=NE6544KWVOUNMKMSARMSJELWHKYGWFHMOFANGJ7MSWNV3EEO4JMA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.707863706 (536h8m9.578946741s) [noise=true , address=45.91.116.11:4118 , piece_id=UDYUJETC735R5WWPE57TRP54BHLL7DTYCYPUNJGZ7RFPUHXF4WYQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.708961978 (536h8m9.580045013s) [noise=true , address=145.79.227.15:4874 , piece_id=N24MZCXSRSPNHTEJSSNTMAW6DU3EUXQPQOP2P5LAEQNMJS6GQREQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.709447032 (536h8m9.580530077s) [noise=true , address=77.90.31.11:4553 , piece_id=WRVJKWKRWIOE3IZ2WW3G5IUGZSA3XBFBTB2UUEDUNBZKI2KDTBQA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.709959948 (536h8m9.581042993s) [noise=true , address=77.90.31.19:4844 , piece_id=3YYYDVPMQSMCHVR2KZONCYIAGEBYFQMOECJQDKEY4XUYOIXZJCCQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.710579397 (536h8m9.581662452s) [noise=true , address=145.79.227.7:4293 , piece_id=M7YGTUL6NE2OLIZYSMF4KDE27HIIYBWIZEOQFUK3Q34HZCVY42UA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.711623495 (536h8m9.582706541s) [noise=true , address=145.79.227.50:4375 , piece_id=7Q5LXWSZCBNCFT6MZLWC2Y5ZYTZ4AGUS5YDCUMWZKGPGRBUYGODA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.713574480 (536h8m9.584657515s) [noise=true , address=77.90.31.16:4735 , piece_id=YWX7IWYYWS5KHGIXB2GUDWLLR4VKKHNIWMSEM3NPVEA6XPDEFHIQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.714036039 (536h8m9.585119084s) [noise=true , address=145.79.227.3:4150 , piece_id=ZOHJ4EEYJXBHHGXGR2ZRFZU2XVW54RE4ZZK7VY5HPJJY5YQL4SEQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.715228480 (536h8m9.586311515s) [noise=true , address=45.91.116.5:4299 , piece_id=653BZ7QJ4QS2FASQIUHLZMIFSEJ7Q54JQ4LHOAZRMQRCBROMORDA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.716142671 (536h8m9.587225716s) [noise=true , address=45.91.116.8:4048 , piece_id=Y5ERSOIELOPSTMS7KAGXATGFLFDU5MHCVYUXV6DB3LO5XLIA4BLQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.716790665 (536h8m9.5878737s) [noise=true , address=145.79.227.5:4223 , piece_id=SB2LII7QIPLOPWBHIHMWUE4X3FKW5C3UZZ23TL2UA7KMXTPR6JIQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.718649134 (536h8m9.589732169s) [noise=true , address=145.79.227.32:4836 , piece_id=A7WCN4AN645HPU3O3U5GX57FFVJKGTWNYIP3IRITJXFJV2WSVIVA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.720527691 (536h8m9.591610736s) [noise=true , address=145.79.227.52:4447 , piece_id=NMH7HHAI36XMOOT4SWLRFP22V4PMFODYI6XC5TQRQFNVFJGKB4IQ, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.724107828 (536h8m9.595190863s) [noise=true , address=145.79.227.11:4730 , piece_id=MYVD2VI5SSWEHKFGL246TYSOF6GWTKV7KPDUHDTTKYDAGPFUFVXA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:09.729567955 (536h8m9.60065099s) [noise=true , address=145.79.227.36:4517 , piece_id=XT3WVKQBKI326NGHAPF6UI3DKYIHHROQM6F52LIWWOUEKCCAXCMA, seg_pos={0 0}, upload=470202]: Uploading piece... Dec 27 23:37:11.436224761 (536h8m11.307307816s) [upload=470202]: Next returned nil segment Dec 27 23:37:11.436238377 (536h8m11.307321412s) [upload=470202]: Waiting for error group managing segments... Dec 27 23:37:11.711936292 (536h8m11.583019327s) [noise=true , address=145.79.227.15:4874 , piece_id=N24MZCXSRSPNHTEJSSNTMAW6DU3EUXQPQOP2P5LAEQNMJS6GQREQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.723417453 (536h8m11.594500498s) [noise=true , address=145.79.227.32:4836 , piece_id=A7WCN4AN645HPU3O3U5GX57FFVJKGTWNYIP3IRITJXFJV2WSVIVA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.727389777 (536h8m11.598472822s) [noise=true , address=145.79.227.11:4730 , piece_id=MYVD2VI5SSWEHKFGL246TYSOF6GWTKV7KPDUHDTTKYDAGPFUFVXA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.730891145 (536h8m11.60197418s) [noise=true , address=145.79.227.4:4187 , piece_id=NE6544KWVOUNMKMSARMSJELWHKYGWFHMOFANGJ7MSWNV3EEO4JMA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.734532098 (536h8m11.605615133s) [noise=true , address=145.79.227.52:4447 , piece_id=NMH7HHAI36XMOOT4SWLRFP22V4PMFODYI6XC5TQRQFNVFJGKB4IQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.735505420 (536h8m11.606588456s) [noise=true , address=145.79.227.7:4293 , piece_id=M7YGTUL6NE2OLIZYSMF4KDE27HIIYBWIZEOQFUK3Q34HZCVY42UA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.735992227 (536h8m11.607075273s) [noise=true , address=145.79.227.36:4517 , piece_id=XT3WVKQBKI326NGHAPF6UI3DKYIHHROQM6F52LIWWOUEKCCAXCMA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.739683547 (536h8m11.610766582s) [noise=true , address=145.79.227.3:4150 , piece_id=ZOHJ4EEYJXBHHGXGR2ZRFZU2XVW54RE4ZZK7VY5HPJJY5YQL4SEQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.746473637 (536h8m11.617556682s) [noise=true , address=145.79.227.5:4223 , piece_id=SB2LII7QIPLOPWBHIHMWUE4X3FKW5C3UZZ23TL2UA7KMXTPR6JIQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.748267662 (536h8m11.619350697s) [noise=true , address=145.79.227.50:4375 , piece_id=7Q5LXWSZCBNCFT6MZLWC2Y5ZYTZ4AGUS5YDCUMWZKGPGRBUYGODA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.909048586 (536h8m11.780131621s) [noise=true , address=77.90.31.19:4844 , piece_id=3YYYDVPMQSMCHVR2KZONCYIAGEBYFQMOECJQDKEY4XUYOIXZJCCQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.969787924 (536h8m11.840870959s) [noise=true , address=45.91.116.11:4118 , piece_id=UDYUJETC735R5WWPE57TRP54BHLL7DTYCYPUNJGZ7RFPUHXF4WYQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.981975691 (536h8m11.853058726s) [noise=true , address=77.90.31.16:4735 , piece_id=YWX7IWYYWS5KHGIXB2GUDWLLR4VKKHNIWMSEM3NPVEA6XPDEFHIQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: <nil> Dec 27 23:37:11.981992322 (536h8m11.853075357s) [seg_pos={0 0}, upload=470202]: Segment reached optimal threshold of 13 pieces. Dec 27 23:37:11.982300379 (536h8m11.853383424s) [noise=true , address=45.91.116.8:4048 , piece_id=Y5ERSOIELOPSTMS7KAGXATGFLFDU5MHCVYUXV6DB3LO5XLIA4BLQ, seg_pos={0 0}, upload=470202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:36568->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:36568->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36568->45.91.116.8:4048: use of closed network connection Dec 27 23:37:11.982310097 (536h8m11.853393142s) [noise=true , address=45.91.116.8:4048 , piece_id=Y5ERSOIELOPSTMS7KAGXATGFLFDU5MHCVYUXV6DB3LO5XLIA4BLQ, seg_pos={0 0}, upload=470202]: Long tail cancellation due to optimal threshold Dec 27 23:37:11.982579100 (536h8m11.853662145s) [noise=true , address=45.91.116.5:4299 , piece_id=653BZ7QJ4QS2FASQIUHLZMIFSEJ7Q54JQ4LHOAZRMQRCBROMORDA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX): protocol: expected piece hash; context canceled; EOF Dec 27 23:37:11.982587776 (536h8m11.853670811s) [noise=true , address=45.91.116.5:4299 , piece_id=653BZ7QJ4QS2FASQIUHLZMIFSEJ7Q54JQ4LHOAZRMQRCBROMORDA, seg_pos={0 0}, upload=470202]: Long tail cancellation due to optimal threshold Dec 27 23:37:11.982662508 (536h8m11.853745553s) [noise=true , address=77.90.31.11:4553 , piece_id=WRVJKWKRWIOE3IZ2WW3G5IUGZSA3XBFBTB2UUEDUNBZKI2KDTBQA, seg_pos={0 0}, upload=470202]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:33516->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:33516->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33516->77.90.31.11:4553: use of closed network connection Dec 27 23:37:11.982673981 (536h8m11.853757026s) [noise=true , address=77.90.31.11:4553 , piece_id=WRVJKWKRWIOE3IZ2WW3G5IUGZSA3XBFBTB2UUEDUNBZKI2KDTBQA, seg_pos={0 0}, upload=470202]: Long tail cancellation due to optimal threshold Dec 27 23:37:11.982696844 (536h8m11.853779879s) [seg_pos={0 0}, upload=470202]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:37:11.982788629 (536h8m11.853871664s) [upload=470202]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:37:11.982795271 (536h8m11.853878306s) [upload=470202]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:37:12.273683530 (536h8m12.144766565s) [upload=470202]: Done uploading segments... Dec 27 23:37:12.273704841 (536h8m12.144787876s) [upload=470202]: Upload finished. err: <nil> Duration: 30.002913452s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............X..........................................X..] duration: 3.566783463s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:37:29 UTC) orbiter warning: context deadline exceeded Duration: 3.56684461s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............X..........................................X.] duration: 3.462920733s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:36:20 UTC) orbiter warning: context deadline exceeded Duration: 3.462948685s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.....X..........................XXX.....XX.XX.XXXX.XXXXXXX.] duration: 12.491438193s trace: https://jaeger.dev.storj.io/trace/57cdf4197a36f7b0 FAIL (2025-12-27 23:36:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: 5XL467CQMZO3IJCVDQJ6Q5PIKKRKSRDYDV6Q5YVOGUVUHOROKVEQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 4FG4USMFRXW66P2SD5HJGPPZKRSMOMPS3Q6DOS7ZEFOLDG3VNAIQ) context deadline exceeded ======LOGS====== Dec 27 23:35:02.063282745 (536h6m1.934365801s) [upload=470172]: Starting upload Dec 27 23:35:02.063687266 (536h6m1.934770331s) [upload=470172]: Done starting upload Dec 27 23:35:02.063761447 (536h6m1.934844492s) [upload=470172]: Uploading segments... Dec 27 23:35:02.063977458 (536h6m1.935060503s) [upload=470172]: Got next segment. Inline: false Dec 27 23:35:02.064053042 (536h6m1.935136107s) [upload=470172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:35:02.064061158 (536h6m1.935144213s) [upload=470172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:35:02.316454005 (536h6m2.187537061s) [seg_pos={0 0}, upload=470172]: Begin upload segment... Dec 27 23:35:02.316523979 (536h6m2.187607024s) [seg_pos={0 0}, upload=470172]: Done begin upload segment. Dec 27 23:35:02.316561620 (536h6m2.187644665s) [noise=true , address=145.79.227.9:4365 , piece_id=B5R2TET7ZJDIU3WKEVO6VNWJNLXGID4T2IMYBALPUJZRZHOOQGPA, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.316610063 (536h6m2.187693108s) [noise=true , address=45.91.116.14:4191 , piece_id=5XL467CQMZO3IJCVDQJ6Q5PIKKRKSRDYDV6Q5YVOGUVUHOROKVEQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.317142877 (536h6m2.188225922s) [noise=true , address=45.91.116.8:4045 , piece_id=4FG4USMFRXW66P2SD5HJGPPZKRSMOMPS3Q6DOS7ZEFOLDG3VNAIQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.317578506 (536h6m2.188661551s) [noise=true , address=45.91.116.4:4937 , piece_id=ZJYLG2SQ3ZGO4UQGBTBAU7KGZ5JXN3WTDRZRFSYBZK6N3SEMI7GQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.318050616 (536h6m2.189133661s) [noise=true , address=77.90.31.17:4770 , piece_id=WBEXRLZJGL6GKPWKTG4S7FZRITMXBUXE2PRXG37VCSVDHMKN5EYQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.318557722 (536h6m2.189640767s) [noise=true , address=145.79.227.15:4874 , piece_id=QWM2YGMXB7G3HMRVAOOTTGYYBVSIE5HPLRKW6HAO5BFQP7YIRJTQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.319046362 (536h6m2.190129407s) [noise=true , address=145.79.227.19:4079 , piece_id=5IRODNM3YBLMFH5FDCDV3GWPJJ3BCQNNNIAR27CCUYL7WOA7XKBQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.319486319 (536h6m2.190569364s) [noise=true , address=145.79.227.35:4482 , piece_id=UV7ZTMPIHBNFE2DDRTU7IX2JY3XMZ45ANDPUTKUHXQF3DFRJKIWQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.319953479 (536h6m2.191036525s) [noise=true , address=145.79.227.45:4230 , piece_id=7XUQEN3GU35SD42R6LHUVNX7ILD4TWF6HTOQOCAMYSNY2JRWDQ4Q, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.323085804 (536h6m2.194168879s) [noise=true , address=45.91.116.10:4081 , piece_id=Z6YD3ATY37RPM2B64AY3DVMYY3ETW5UYDX7AGHXNLJ5SMHVG3VVQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.323399121 (536h6m2.194482166s) [noise=true , address=145.79.227.42:4121 , piece_id=X37KJWPGFKRRFMMWNZB57DWBLEB4RTBMCV7SQ7SP4QJUS2YOQTKA, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.324241965 (536h6m2.19532501s) [noise=true , address=145.79.227.11:4730 , piece_id=S7QKB45GFSHJRB2PI5DIKRAPQMNPDLEC7U6DEPLFFIH6FN6M6L6A, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.325915502 (536h6m2.196998547s) [noise=true , address=145.79.227.39:4013 , piece_id=4NXHHFMVV3JJX4LP7GT5PO6BK4JHZHTPEIVYY35LNHS4CRBYG2LQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.327865876 (536h6m2.198948921s) [noise=true , address=145.79.227.40:4049 , piece_id=DWRIVUEGJSU7I4BWEYQIHNONAOYA5Y7XQCC6KNONVNYMLPMZIXIQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.329494768 (536h6m2.200577813s) [noise=true , address=45.91.116.12:4228 , piece_id=R7EFTI5XHM5VLSM75YFTFTZUJWVKZJWOJILJBMRT4LZ425BNQY6Q, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.331331976 (536h6m2.202415021s) [noise=true , address=145.79.227.16:4911 , piece_id=OCYR4NWJJT4U44XZRUZCOOR6GEZ5QS4AE7ALR32VMZYPCJ7OATQQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:03.937771813 (536h6m3.808854858s) [upload=470172]: Next returned nil segment Dec 27 23:35:03.937787382 (536h6m3.808870417s) [upload=470172]: Waiting for error group managing segments... Dec 27 23:35:04.323926656 (536h6m4.195009701s) [noise=true , address=145.79.227.15:4874 , piece_id=QWM2YGMXB7G3HMRVAOOTTGYYBVSIE5HPLRKW6HAO5BFQP7YIRJTQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.325169151 (536h6m4.196252226s) [noise=true , address=145.79.227.39:4013 , piece_id=4NXHHFMVV3JJX4LP7GT5PO6BK4JHZHTPEIVYY35LNHS4CRBYG2LQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.333176078 (536h6m4.204259123s) [noise=true , address=145.79.227.16:4911 , piece_id=OCYR4NWJJT4U44XZRUZCOOR6GEZ5QS4AE7ALR32VMZYPCJ7OATQQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.341676575 (536h6m4.21275963s) [noise=true , address=145.79.227.11:4730 , piece_id=S7QKB45GFSHJRB2PI5DIKRAPQMNPDLEC7U6DEPLFFIH6FN6M6L6A, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.343279537 (536h6m4.214362582s) [noise=true , address=145.79.227.42:4121 , piece_id=X37KJWPGFKRRFMMWNZB57DWBLEB4RTBMCV7SQ7SP4QJUS2YOQTKA, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.344045576 (536h6m4.215128621s) [noise=true , address=145.79.227.19:4079 , piece_id=5IRODNM3YBLMFH5FDCDV3GWPJJ3BCQNNNIAR27CCUYL7WOA7XKBQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.353553100 (536h6m4.224636145s) [noise=true , address=145.79.227.45:4230 , piece_id=7XUQEN3GU35SD42R6LHUVNX7ILD4TWF6HTOQOCAMYSNY2JRWDQ4Q, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.357024419 (536h6m4.228107464s) [noise=true , address=145.79.227.9:4365 , piece_id=B5R2TET7ZJDIU3WKEVO6VNWJNLXGID4T2IMYBALPUJZRZHOOQGPA, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.360287533 (536h6m4.231370578s) [noise=true , address=145.79.227.40:4049 , piece_id=DWRIVUEGJSU7I4BWEYQIHNONAOYA5Y7XQCC6KNONVNYMLPMZIXIQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.442082894 (536h6m4.313165939s) [noise=true , address=145.79.227.35:4482 , piece_id=UV7ZTMPIHBNFE2DDRTU7IX2JY3XMZ45ANDPUTKUHXQF3DFRJKIWQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.556881410 (536h6m4.427964465s) [noise=true , address=45.91.116.4:4937 , piece_id=ZJYLG2SQ3ZGO4UQGBTBAU7KGZ5JXN3WTDRZRFSYBZK6N3SEMI7GQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.558533054 (536h6m4.429616099s) [noise=true , address=45.91.116.8:4045 , piece_id=4FG4USMFRXW66P2SD5HJGPPZKRSMOMPS3Q6DOS7ZEFOLDG3VNAIQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.584952842 (536h6m4.456035897s) [noise=true , address=45.91.116.14:4191 , piece_id=5XL467CQMZO3IJCVDQJ6Q5PIKKRKSRDYDV6Q5YVOGUVUHOROKVEQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.584982549 (536h6m4.456065594s) [seg_pos={0 0}, upload=470172]: Segment reached optimal threshold of 13 pieces. Dec 27 23:35:04.585259316 (536h6m4.456342361s) [noise=true , address=45.91.116.12:4228 , piece_id=R7EFTI5XHM5VLSM75YFTFTZUJWVKZJWOJILJBMRT4LZ425BNQY6Q, seg_pos={0 0}, upload=470172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 27 23:35:04.585270537 (536h6m4.456353582s) [noise=true , address=45.91.116.12:4228 , piece_id=R7EFTI5XHM5VLSM75YFTFTZUJWVKZJWOJILJBMRT4LZ425BNQY6Q, seg_pos={0 0}, upload=470172]: Long tail cancellation due to optimal threshold Dec 27 23:35:04.585673565 (536h6m4.45675661s) [noise=true , address=45.91.116.10:4081 , piece_id=Z6YD3ATY37RPM2B64AY3DVMYY3ETW5UYDX7AGHXNLJ5SMHVG3VVQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 27 23:35:04.585685006 (536h6m4.456768051s) [noise=true , address=45.91.116.10:4081 , piece_id=Z6YD3ATY37RPM2B64AY3DVMYY3ETW5UYDX7AGHXNLJ5SMHVG3VVQ, seg_pos={0 0}, upload=470172]: Long tail cancellation due to optimal threshold Dec 27 23:35:04.585898262 (536h6m4.456981307s) [noise=true , address=77.90.31.17:4770 , piece_id=WBEXRLZJGL6GKPWKTG4S7FZRITMXBUXE2PRXG37VCSVDHMKN5EYQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 27 23:35:04.585907831 (536h6m4.456990876s) [noise=true , address=77.90.31.17:4770 , piece_id=WBEXRLZJGL6GKPWKTG4S7FZRITMXBUXE2PRXG37VCSVDHMKN5EYQ, seg_pos={0 0}, upload=470172]: Long tail cancellation due to optimal threshold Dec 27 23:35:04.585935122 (536h6m4.457018168s) [seg_pos={0 0}, upload=470172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:35:04.586513544 (536h6m4.457596589s) [upload=470172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:35:04.586522972 (536h6m4.457606017s) [upload=470172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:35:04.880514842 (536h6m4.751597887s) [upload=470172]: Done uploading segments... Dec 27 23:35:04.880536242 (536h6m4.751619287s) [upload=470172]: Upload finished. err: <nil> Duration: 12.491462591s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................X..........................................X] duration: 5.001235379s trace: https://jaeger.dev.storj.io/trace/1b11fe35ac5cdbcb FAIL (2025-12-27 23:35:33 UTC) orbiter warning: context deadline exceeded Duration: 5.001256598s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.....X..........................XXX.....XX.XX.XXXX.XXXXXXX] duration: 30.004124827s trace: https://jaeger.dev.storj.io/trace/57cdf4197a36f7b0 FAIL (2025-12-27 23:35:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: 5XL467CQMZO3IJCVDQJ6Q5PIKKRKSRDYDV6Q5YVOGUVUHOROKVEQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: 4FG4USMFRXW66P2SD5HJGPPZKRSMOMPS3Q6DOS7ZEFOLDG3VNAIQ) context deadline exceeded ======LOGS====== Dec 27 23:35:02.063282745 (536h6m1.934365801s) [upload=470172]: Starting upload Dec 27 23:35:02.063687266 (536h6m1.934770331s) [upload=470172]: Done starting upload Dec 27 23:35:02.063761447 (536h6m1.934844492s) [upload=470172]: Uploading segments... Dec 27 23:35:02.063977458 (536h6m1.935060503s) [upload=470172]: Got next segment. Inline: false Dec 27 23:35:02.064053042 (536h6m1.935136107s) [upload=470172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:35:02.064061158 (536h6m1.935144213s) [upload=470172]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:35:02.316454005 (536h6m2.187537061s) [seg_pos={0 0}, upload=470172]: Begin upload segment... Dec 27 23:35:02.316523979 (536h6m2.187607024s) [seg_pos={0 0}, upload=470172]: Done begin upload segment. Dec 27 23:35:02.316561620 (536h6m2.187644665s) [noise=true , address=145.79.227.9:4365 , piece_id=B5R2TET7ZJDIU3WKEVO6VNWJNLXGID4T2IMYBALPUJZRZHOOQGPA, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.316610063 (536h6m2.187693108s) [noise=true , address=45.91.116.14:4191 , piece_id=5XL467CQMZO3IJCVDQJ6Q5PIKKRKSRDYDV6Q5YVOGUVUHOROKVEQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.317142877 (536h6m2.188225922s) [noise=true , address=45.91.116.8:4045 , piece_id=4FG4USMFRXW66P2SD5HJGPPZKRSMOMPS3Q6DOS7ZEFOLDG3VNAIQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.317578506 (536h6m2.188661551s) [noise=true , address=45.91.116.4:4937 , piece_id=ZJYLG2SQ3ZGO4UQGBTBAU7KGZ5JXN3WTDRZRFSYBZK6N3SEMI7GQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.318050616 (536h6m2.189133661s) [noise=true , address=77.90.31.17:4770 , piece_id=WBEXRLZJGL6GKPWKTG4S7FZRITMXBUXE2PRXG37VCSVDHMKN5EYQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.318557722 (536h6m2.189640767s) [noise=true , address=145.79.227.15:4874 , piece_id=QWM2YGMXB7G3HMRVAOOTTGYYBVSIE5HPLRKW6HAO5BFQP7YIRJTQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.319046362 (536h6m2.190129407s) [noise=true , address=145.79.227.19:4079 , piece_id=5IRODNM3YBLMFH5FDCDV3GWPJJ3BCQNNNIAR27CCUYL7WOA7XKBQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.319486319 (536h6m2.190569364s) [noise=true , address=145.79.227.35:4482 , piece_id=UV7ZTMPIHBNFE2DDRTU7IX2JY3XMZ45ANDPUTKUHXQF3DFRJKIWQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.319953479 (536h6m2.191036525s) [noise=true , address=145.79.227.45:4230 , piece_id=7XUQEN3GU35SD42R6LHUVNX7ILD4TWF6HTOQOCAMYSNY2JRWDQ4Q, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.323085804 (536h6m2.194168879s) [noise=true , address=45.91.116.10:4081 , piece_id=Z6YD3ATY37RPM2B64AY3DVMYY3ETW5UYDX7AGHXNLJ5SMHVG3VVQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.323399121 (536h6m2.194482166s) [noise=true , address=145.79.227.42:4121 , piece_id=X37KJWPGFKRRFMMWNZB57DWBLEB4RTBMCV7SQ7SP4QJUS2YOQTKA, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.324241965 (536h6m2.19532501s) [noise=true , address=145.79.227.11:4730 , piece_id=S7QKB45GFSHJRB2PI5DIKRAPQMNPDLEC7U6DEPLFFIH6FN6M6L6A, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.325915502 (536h6m2.196998547s) [noise=true , address=145.79.227.39:4013 , piece_id=4NXHHFMVV3JJX4LP7GT5PO6BK4JHZHTPEIVYY35LNHS4CRBYG2LQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.327865876 (536h6m2.198948921s) [noise=true , address=145.79.227.40:4049 , piece_id=DWRIVUEGJSU7I4BWEYQIHNONAOYA5Y7XQCC6KNONVNYMLPMZIXIQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.329494768 (536h6m2.200577813s) [noise=true , address=45.91.116.12:4228 , piece_id=R7EFTI5XHM5VLSM75YFTFTZUJWVKZJWOJILJBMRT4LZ425BNQY6Q, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:02.331331976 (536h6m2.202415021s) [noise=true , address=145.79.227.16:4911 , piece_id=OCYR4NWJJT4U44XZRUZCOOR6GEZ5QS4AE7ALR32VMZYPCJ7OATQQ, seg_pos={0 0}, upload=470172]: Uploading piece... Dec 27 23:35:03.937771813 (536h6m3.808854858s) [upload=470172]: Next returned nil segment Dec 27 23:35:03.937787382 (536h6m3.808870417s) [upload=470172]: Waiting for error group managing segments... Dec 27 23:35:04.323926656 (536h6m4.195009701s) [noise=true , address=145.79.227.15:4874 , piece_id=QWM2YGMXB7G3HMRVAOOTTGYYBVSIE5HPLRKW6HAO5BFQP7YIRJTQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.325169151 (536h6m4.196252226s) [noise=true , address=145.79.227.39:4013 , piece_id=4NXHHFMVV3JJX4LP7GT5PO6BK4JHZHTPEIVYY35LNHS4CRBYG2LQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.333176078 (536h6m4.204259123s) [noise=true , address=145.79.227.16:4911 , piece_id=OCYR4NWJJT4U44XZRUZCOOR6GEZ5QS4AE7ALR32VMZYPCJ7OATQQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.341676575 (536h6m4.21275963s) [noise=true , address=145.79.227.11:4730 , piece_id=S7QKB45GFSHJRB2PI5DIKRAPQMNPDLEC7U6DEPLFFIH6FN6M6L6A, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.343279537 (536h6m4.214362582s) [noise=true , address=145.79.227.42:4121 , piece_id=X37KJWPGFKRRFMMWNZB57DWBLEB4RTBMCV7SQ7SP4QJUS2YOQTKA, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.344045576 (536h6m4.215128621s) [noise=true , address=145.79.227.19:4079 , piece_id=5IRODNM3YBLMFH5FDCDV3GWPJJ3BCQNNNIAR27CCUYL7WOA7XKBQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.353553100 (536h6m4.224636145s) [noise=true , address=145.79.227.45:4230 , piece_id=7XUQEN3GU35SD42R6LHUVNX7ILD4TWF6HTOQOCAMYSNY2JRWDQ4Q, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.357024419 (536h6m4.228107464s) [noise=true , address=145.79.227.9:4365 , piece_id=B5R2TET7ZJDIU3WKEVO6VNWJNLXGID4T2IMYBALPUJZRZHOOQGPA, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.360287533 (536h6m4.231370578s) [noise=true , address=145.79.227.40:4049 , piece_id=DWRIVUEGJSU7I4BWEYQIHNONAOYA5Y7XQCC6KNONVNYMLPMZIXIQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.442082894 (536h6m4.313165939s) [noise=true , address=145.79.227.35:4482 , piece_id=UV7ZTMPIHBNFE2DDRTU7IX2JY3XMZ45ANDPUTKUHXQF3DFRJKIWQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.556881410 (536h6m4.427964465s) [noise=true , address=45.91.116.4:4937 , piece_id=ZJYLG2SQ3ZGO4UQGBTBAU7KGZ5JXN3WTDRZRFSYBZK6N3SEMI7GQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.558533054 (536h6m4.429616099s) [noise=true , address=45.91.116.8:4045 , piece_id=4FG4USMFRXW66P2SD5HJGPPZKRSMOMPS3Q6DOS7ZEFOLDG3VNAIQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.584952842 (536h6m4.456035897s) [noise=true , address=45.91.116.14:4191 , piece_id=5XL467CQMZO3IJCVDQJ6Q5PIKKRKSRDYDV6Q5YVOGUVUHOROKVEQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: <nil> Dec 27 23:35:04.584982549 (536h6m4.456065594s) [seg_pos={0 0}, upload=470172]: Segment reached optimal threshold of 13 pieces. Dec 27 23:35:04.585259316 (536h6m4.456342361s) [noise=true , address=45.91.116.12:4228 , piece_id=R7EFTI5XHM5VLSM75YFTFTZUJWVKZJWOJILJBMRT4LZ425BNQY6Q, seg_pos={0 0}, upload=470172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 27 23:35:04.585270537 (536h6m4.456353582s) [noise=true , address=45.91.116.12:4228 , piece_id=R7EFTI5XHM5VLSM75YFTFTZUJWVKZJWOJILJBMRT4LZ425BNQY6Q, seg_pos={0 0}, upload=470172]: Long tail cancellation due to optimal threshold Dec 27 23:35:04.585673565 (536h6m4.45675661s) [noise=true , address=45.91.116.10:4081 , piece_id=Z6YD3ATY37RPM2B64AY3DVMYY3ETW5UYDX7AGHXNLJ5SMHVG3VVQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 27 23:35:04.585685006 (536h6m4.456768051s) [noise=true , address=45.91.116.10:4081 , piece_id=Z6YD3ATY37RPM2B64AY3DVMYY3ETW5UYDX7AGHXNLJ5SMHVG3VVQ, seg_pos={0 0}, upload=470172]: Long tail cancellation due to optimal threshold Dec 27 23:35:04.585898262 (536h6m4.456981307s) [noise=true , address=77.90.31.17:4770 , piece_id=WBEXRLZJGL6GKPWKTG4S7FZRITMXBUXE2PRXG37VCSVDHMKN5EYQ, seg_pos={0 0}, upload=470172]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: expected piece hash; context canceled; EOF Dec 27 23:35:04.585907831 (536h6m4.456990876s) [noise=true , address=77.90.31.17:4770 , piece_id=WBEXRLZJGL6GKPWKTG4S7FZRITMXBUXE2PRXG37VCSVDHMKN5EYQ, seg_pos={0 0}, upload=470172]: Long tail cancellation due to optimal threshold Dec 27 23:35:04.585935122 (536h6m4.457018168s) [seg_pos={0 0}, upload=470172]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:35:04.586513544 (536h6m4.457596589s) [upload=470172]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:35:04.586522972 (536h6m4.457606017s) [upload=470172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:35:04.880514842 (536h6m4.751597887s) [upload=470172]: Done uploading segments... Dec 27 23:35:04.880536242 (536h6m4.751619287s) [upload=470172]: Upload finished. err: <nil> Duration: 30.004152578s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.540446934s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:35:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.540462716s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X.....X..........................XXX.....XX.XX.XXXX.XXXXXX] duration: 30.003080813s trace: https://jaeger.dev.storj.io/trace/aee46117bf9d3cb FAIL (2025-12-27 23:34:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: FQVH447TMUDATMIKRGGJOMSTOCYSMVMRHHB2LCJN7L3FFOIXMARA) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: UG4RFZVVTDL2PVEUX4XBEDB6JGWDNH7LOFBVIIU53R6XKHZAP7AQ) context deadline exceeded ======LOGS====== 5m19.222024788s) [upload=470159]: Uploading segments... Dec 27 23:34:19.351062925 (536h5m19.22214596s) [upload=470159]: Got next segment. Inline: false Dec 27 23:34:19.351101177 (536h5m19.222184212s) [upload=470159]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:34:19.351107139 (536h5m19.222190184s) [upload=470159]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:34:19.595038313 (536h5m19.466121358s) [seg_pos={0 0}, upload=470159]: Begin upload segment... Dec 27 23:34:19.595093569 (536h5m19.466176614s) [seg_pos={0 0}, upload=470159]: Done begin upload segment. Dec 27 23:34:19.595136520 (536h5m19.466219556s) [noise=true , address=145.79.227.41:4086 , piece_id=7KZRANXQLK3B2ZZQQF3QN3ZA3YW43BJMAUJTK2PA72IZ6RENHOBA, seg_pos={0 0}, upload=470159]: Uploading piece... Dec 27 23:34:19.595154505 (536h5m19.46623758s) [noise=true , address=145.79.227.32:4836 , piece_id=NU75ORZE7ADQOKYE65W74DK5PIVTU3HRHGMI4OWCIY6SFIUPPNXQ, seg_pos={0 0}, upload=470159]: Uploading piece... Dec 27 23:34:19.595660448 (536h5m19.466743483s) [noise=true , address=77.90.31.21:4877 , piece_id=M4AYWUGC5UQPU5ZYHGV4GO3HTR4IRSO3XSFGMNVJINKFX4AG4BNQ, seg_pos={0 0}, upload=470159]: Uploading piece... Dec 27 23:34:19.595818089 (536h5m19.466901124s) [noise=true , address=145.79.227.51:4410 , piece_id=4Y3CEYIXJCZ3ACUOUC6JEFRWVS5GGDEW7ZBIZKB33FVBH4WJSS7A, seg_pos={0 0}, upload=470159]: Uploading piece... Dec 27 23:34:19.596040101 (536h5m19.467123136s) [noise=true , address=45.91.116.12:4228 , piece_id=FQVH447TMUDATMIKRGGJOMSTOCYSMVMRHHB2LCJN7L3FFOIXMARA, seg_pos={0 0}, upload=470159]: Uploading piece... Dec 27 23:34:19.596197291 (536h5m19.467280326s) [noise=true , address=77.90.31.19:4844 , piece_id=2ZJ5AJ7SPZ4YK6T5EQ3NS5LGZ66HKC6TEJWL2ZDCU7VSD6WEAXMA, seg_pos={0 0}, upload=470159]: Uploading piece... Dec 27 23:34:19.596410557 (536h5m19.467493602s) [noise=true , address=145.79.227.45:4229 , piece_id=DVETRT5LUBPVZXOWZPDV7J6IEVCY4ROPIBI25RE5E43Q4KA5TJMA, seg_pos={0 0}, upload=470159]: Uploading piece... Dec 27 23:34:19.596553739 (536h5m19.467636774s) [noise=true , address=145.79.227.49:4339 , piece_id=XUGT2F2W4R4QMJNT5IPCY7ZYDNKONBQCHXNDYFM6DLW25OKYSXGA, seg_pos={0 0}, upload=470159]: Uploading piece... Dec 27 23:34:19.596787024 (536h5m19.467870059s) [noise=true , address=45.91.116.7:4011 , piece_id=SDSIIZCTBIDSLHTND5K576WRMXKNSQ5R7WDZ6AY7KXYMHOVCEH2Q, seg_pos={0 0}, upload=470159]: Uploading piece... Dec 27 23:34:19.596883918 (536h5m19.467966953s) [noise=true , address=77.90.31.13:4629 , piece_id=3CJSAHA7LYXB45SIOYKK3BXVKTKHGYTGAH57GBASRDTHY3VIX3WQ, seg_pos={0 0}, upload=470159]: Uploading piece... Dec 27 23:34:19.597177197 (536h5m19.468260232s) [noise=true , address=145.79.227.42:4123 , piece_id=C5RQWOPQQXA7P3LEPD457MQ2F537R2FC2TRLFISF3BWKLETWT7XQ, seg_pos={0 0}, upload=470159]: Uploading piece... Dec 27 23:34:19.597184080 (536h5m19.468267125s) [noise=true , address=145.79.227.25:4618 , piece_id=3IEC2BIXCHUDBSZ2GIDE3GB6TDVPDNKJBFT5LGEURHYM5QZO5ZEA, seg_pos={0 0}, upload=470159]: Uploading piece... Dec 27 23:34:19.597519108 (536h5m19.468602143s) [noise=true , address=45.91.116.11:4118 , piece_id=UG4RFZVVTDL2PVEUX4XBEDB6JGWDNH7LOFBVIIU53R6XKHZAP7AQ, seg_pos={0 0}, upload=470159]: Uploading piece... Dec 27 23:34:19.597539095 (536h5m19.46862213s) [noise=true , address=77.90.31.18:4808 , piece_id=BBU5UVPT6LYCIBSM6PVUPY6IPK4BJCXDB746OAOWPYSPW2GYVFDA, seg_pos={0 0}, upload=470159]: Uploading piece... Dec 27 23:34:19.597911284 (536h5m19.468994329s) [noise=true , address=145.79.227.13:4804 , piece_id=SDQMLBCSPHPOWHTLPXITZETRQMGDBT7DNHVP5TIMRULRC2ZBULRQ, seg_pos={0 0}, upload=470159]: Uploading piece... Dec 27 23:34:19.599281112 (536h5m19.470364157s) [noise=true , address=45.91.116.5:4299 , piece_id=RZ7U4R57NFQXBF7LLYROOPLFT6YYY4K3TMBF24RPN6GQIJXIDBGQ, seg_pos={0 0}, upload=470159]: Uploading piece... Dec 27 23:34:21.307843428 (536h5m21.178926463s) [upload=470159]: Next returned nil segment Dec 27 23:34:21.307870149 (536h5m21.178953184s) [upload=470159]: Waiting for error group managing segments... Dec 27 23:34:21.593423028 (536h5m21.464506063s) [noise=true , address=145.79.227.32:4836 , piece_id=NU75ORZE7ADQOKYE65W74DK5PIVTU3HRHGMI4OWCIY6SFIUPPNXQ, seg_pos={0 0}, upload=470159]: Done uploading piece. err: <nil> Dec 27 23:34:21.596378886 (536h5m21.467461921s) [noise=true , address=145.79.227.42:4123 , piece_id=C5RQWOPQQXA7P3LEPD457MQ2F537R2FC2TRLFISF3BWKLETWT7XQ, seg_pos={0 0}, upload=470159]: Done uploading piece. err: <nil> Dec 27 23:34:21.598720735 (536h5m21.46980377s) [noise=true , address=145.79.227.25:4618 , piece_id=3IEC2BIXCHUDBSZ2GIDE3GB6TDVPDNKJBFT5LGEURHYM5QZO5ZEA, seg_pos={0 0}, upload=470159]: Done uploading piece. err: <nil> Dec 27 23:34:21.602478491 (536h5m21.473561536s) [noise=true , address=145.79.227.45:4229 , piece_id=DVETRT5LUBPVZXOWZPDV7J6IEVCY4ROPIBI25RE5E43Q4KA5TJMA, seg_pos={0 0}, upload=470159]: Done uploading piece. err: <nil> Dec 27 23:34:21.606516790 (536h5m21.477599845s) [noise=true , address=145.79.227.13:4804 , piece_id=SDQMLBCSPHPOWHTLPXITZETRQMGDBT7DNHVP5TIMRULRC2ZBULRQ, seg_pos={0 0}, upload=470159]: Done uploading piece. err: <nil> Dec 27 23:34:21.617445310 (536h5m21.488528355s) [noise=true , address=145.79.227.41:4086 , piece_id=7KZRANXQLK3B2ZZQQF3QN3ZA3YW43BJMAUJTK2PA72IZ6RENHOBA, seg_pos={0 0}, upload=470159]: Done uploading piece. err: <nil> Dec 27 23:34:21.825974595 (536h5m21.69705765s) [noise=true , address=45.91.116.7:4011 , piece_id=SDSIIZCTBIDSLHTND5K576WRMXKNSQ5R7WDZ6AY7KXYMHOVCEH2Q, seg_pos={0 0}, upload=470159]: Done uploading piece. err: <nil> Dec 27 23:34:21.828855870 (536h5m21.699938915s) [noise=true , address=77.90.31.19:4844 , piece_id=2ZJ5AJ7SPZ4YK6T5EQ3NS5LGZ66HKC6TEJWL2ZDCU7VSD6WEAXMA, seg_pos={0 0}, upload=470159]: Done uploading piece. err: <nil> Dec 27 23:34:21.863952832 (536h5m21.735035877s) [noise=true , address=45.91.116.12:4228 , piece_id=FQVH447TMUDATMIKRGGJOMSTOCYSMVMRHHB2LCJN7L3FFOIXMARA, seg_pos={0 0}, upload=470159]: Done uploading piece. err: <nil> Dec 27 23:34:21.906428421 (536h5m21.777511466s) [noise=true , address=45.91.116.11:4118 , piece_id=UG4RFZVVTDL2PVEUX4XBEDB6JGWDNH7LOFBVIIU53R6XKHZAP7AQ, seg_pos={0 0}, upload=470159]: Done uploading piece. err: <nil> Dec 27 23:34:21.940682397 (536h5m21.811765432s) [noise=true , address=77.90.31.13:4629 , piece_id=3CJSAHA7LYXB45SIOYKK3BXVKTKHGYTGAH57GBASRDTHY3VIX3WQ, seg_pos={0 0}, upload=470159]: Done uploading piece. err: <nil> Dec 27 23:34:21.969368440 (536h5m21.840451485s) [noise=true , address=45.91.116.5:4299 , piece_id=RZ7U4R57NFQXBF7LLYROOPLFT6YYY4K3TMBF24RPN6GQIJXIDBGQ, seg_pos={0 0}, upload=470159]: Done uploading piece. err: <nil> Dec 27 23:34:22.034020388 (536h5m21.905103423s) [noise=true , address=77.90.31.21:4877 , piece_id=M4AYWUGC5UQPU5ZYHGV4GO3HTR4IRSO3XSFGMNVJINKFX4AG4BNQ, seg_pos={0 0}, upload=470159]: Done uploading piece. err: <nil> Dec 27 23:34:22.034040476 (536h5m21.905123511s) [seg_pos={0 0}, upload=470159]: Segment reached optimal threshold of 13 pieces. Dec 27 23:34:22.034270023 (536h5m21.905353058s) [noise=true , address=145.79.227.51:4410 , piece_id=4Y3CEYIXJCZ3ACUOUC6JEFRWVS5GGDEW7ZBIZKB33FVBH4WJSS7A, seg_pos={0 0}, upload=470159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: write tcp 188.166.225.170:34866->145.79.227.51:4410: use of closed network connection; write tcp 188.166.225.170:34866->145.79.227.51:4410: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34866->145.79.227.51:4410: use of closed network connection Dec 27 23:34:22.034276495 (536h5m21.905359531s) [noise=true , address=145.79.227.51:4410 , piece_id=4Y3CEYIXJCZ3ACUOUC6JEFRWVS5GGDEW7ZBIZKB33FVBH4WJSS7A, seg_pos={0 0}, upload=470159]: Long tail cancellation due to optimal threshold Dec 27 23:34:22.034490473 (536h5m21.905573508s) [noise=true , address=77.90.31.18:4808 , piece_id=BBU5UVPT6LYCIBSM6PVUPY6IPK4BJCXDB746OAOWPYSPW2GYVFDA, seg_pos={0 0}, upload=470159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:43790->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:43790->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43790->77.90.31.18:4808: use of closed network connection Dec 27 23:34:22.034497396 (536h5m21.905580441s) [noise=true , address=77.90.31.18:4808 , piece_id=BBU5UVPT6LYCIBSM6PVUPY6IPK4BJCXDB746OAOWPYSPW2GYVFDA, seg_pos={0 0}, upload=470159]: Long tail cancellation due to optimal threshold Dec 27 23:34:22.034613768 (536h5m21.905696813s) [noise=true , address=145.79.227.49:4339 , piece_id=XUGT2F2W4R4QMJNT5IPCY7ZYDNKONBQCHXNDYFM6DLW25OKYSXGA, seg_pos={0 0}, upload=470159]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 27 23:34:22.034619409 (536h5m21.905702444s) [noise=true , address=145.79.227.49:4339 , piece_id=XUGT2F2W4R4QMJNT5IPCY7ZYDNKONBQCHXNDYFM6DLW25OKYSXGA, seg_pos={0 0}, upload=470159]: Long tail cancellation due to optimal threshold Dec 27 23:34:22.034635869 (536h5m21.905718915s) [seg_pos={0 0}, upload=470159]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:34:22.034709680 (536h5m21.905792715s) [upload=470159]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:34:22.034715101 (536h5m21.905798136s) [upload=470159]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:34:22.323197559 (536h5m22.194280624s) [upload=470159]: Done uploading segments... Dec 27 23:34:22.323221995 (536h5m22.19430503s) [upload=470159]: Upload finished. err: <nil> Duration: 30.003091183s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.531476574s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:34:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.531502745s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X.....X..........................XXX.....XX.XX.XXXX.XXXXX] duration: 30.003541126s trace: https://jaeger.dev.storj.io/trace/29a28da59c19552b FAIL (2025-12-27 23:34:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: DYZ7OQXT4BO2QX3PIBK2NMN3LCKDG5WJOTR5DUH7ZWHBEXY3Y6CA) context deadline exceeded; piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: B6IB7VXP5Q34RKRWRJNTRAPPOYFXD25VB4OVUSUHL7W3PQXSM7CA) context deadline exceeded ======LOGS====== Dec 27 23:33:49.120171715 (536h4m48.99125475s) [upload=470152]: Starting upload Dec 27 23:33:49.120252519 (536h4m48.991335554s) [upload=470152]: Done starting upload Dec 27 23:33:49.120318665 (536h4m48.99140171s) [upload=470152]: Uploading segments... Dec 27 23:33:49.120496283 (536h4m48.991579318s) [upload=470152]: Got next segment. Inline: false Dec 27 23:33:49.120545135 (536h4m48.991628171s) [upload=470152]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:33:49.120550726 (536h4m48.991633761s) [upload=470152]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:33:49.371764379 (536h4m49.242847414s) [seg_pos={0 0}, upload=470152]: Begin upload segment... Dec 27 23:33:49.371805006 (536h4m49.242888041s) [seg_pos={0 0}, upload=470152]: Done begin upload segment. Dec 27 23:33:49.371831436 (536h4m49.242914471s) [noise=true , address=77.90.31.12:4589 , piece_id=FAVB2HA4K346XXAMONOCIDT7IAWRM7PCXJXPKEYNSIGYW7YQEVLA, seg_pos={0 0}, upload=470152]: Uploading piece... Dec 27 23:33:49.371954381 (536h4m49.243037416s) [noise=true , address=45.91.116.4:4940 , piece_id=M55UKXPZ2XB2MEW7B5ZSWW4GT7FEHYWK2NJXSV3IA6H5JZJVPWTQ, seg_pos={0 0}, upload=470152]: Uploading piece... Dec 27 23:33:49.372577286 (536h4m49.243660321s) [noise=true , address=145.79.227.32:4836 , piece_id=HDI3NBNAB3WGFK6DPZVG74UMIBHOYWKWOSPYFLLBL34NHYLMJPLQ, seg_pos={0 0}, upload=470152]: Uploading piece... Dec 27 23:33:49.372937883 (536h4m49.244020918s) [noise=true , address=145.79.227.2:4113 , piece_id=75KW7BRUFS57ZPJQHA6L4F7NHI5OZCBYUACMT46SC6SMACQT3NHQ, seg_pos={0 0}, upload=470152]: Uploading piece... Dec 27 23:33:49.373210994 (536h4m49.244294029s) [noise=true , address=145.79.227.12:4765 , piece_id=7C2KD5PG35IEYDYLJ2RBYAAAR6PGHUFMPB6Y7BF4ND4ISRAR67AQ, seg_pos={0 0}, upload=470152]: Uploading piece... Dec 27 23:33:49.373484063 (536h4m49.244567098s) [noise=true , address=45.91.116.12:4228 , piece_id=B6IB7VXP5Q34RKRWRJNTRAPPOYFXD25VB4OVUSUHL7W3PQXSM7CA, seg_pos={0 0}, upload=470152]: Uploading piece... Dec 27 23:33:49.373754778 (536h4m49.244837814s) [noise=true , address=45.91.116.11:4117 , piece_id=QQCPS2JHRSGK4HPU5XA43WOQILXNVOQOL4YIZCKXBASWZJV6I4JA, seg_pos={0 0}, upload=470152]: Uploading piece... Dec 27 23:33:49.374065069 (536h4m49.245148104s) [noise=true , address=145.79.227.40:4049 , piece_id=NI2W7IRRO4R4VZ3PDTFY4KMQA7LGVVQT42DPWSB3SC2KOTBCEPKA, seg_pos={0 0}, upload=470152]: Uploading piece... Dec 27 23:33:49.375239175 (536h4m49.24632221s) [noise=true , address=145.79.227.48:4303 , piece_id=6UHP65PGTKISETS3VKRM472HFIKW2YWNHK7LQFG6RKDZDVCTST2A, seg_pos={0 0}, upload=470152]: Uploading piece... Dec 27 23:33:49.376358406 (536h4m49.247441441s) [noise=true , address=145.79.227.11:4729 , piece_id=5ZKLEDW2ACLXWU5BNOKZYPTJWTNH4NEUEFURKF6KP4NTBKHZ2GRQ, seg_pos={0 0}, upload=470152]: Uploading piece... Dec 27 23:33:49.376567214 (536h4m49.247650249s) [noise=true , address=145.79.227.42:4121 , piece_id=X5TLQLAGJTNVX7QLTZZEJQ5DIXH24LDOFDFTCR5TULFV4NNAUXXA, seg_pos={0 0}, upload=470152]: Uploading piece... Dec 27 23:33:49.377897297 (536h4m49.248980342s) [noise=true , address=145.79.227.9:4365 , piece_id=H3MS6PAY6OSTMZ53ALUKWWPES52W3BVZ2AA6KBPW3XR5EQTAVFDA, seg_pos={0 0}, upload=470152]: Uploading piece... Dec 27 23:33:49.382338614 (536h4m49.253421659s) [noise=true , address=45.91.116.13:4153 , piece_id=DYZ7OQXT4BO2QX3PIBK2NMN3LCKDG5WJOTR5DUH7ZWHBEXY3Y6CA, seg_pos={0 0}, upload=470152]: Uploading piece... Dec 27 23:33:49.383067442 (536h4m49.254150477s) [noise=true , address=145.79.227.37:4567 , piece_id=PNNETAJYZ7PY6GL2WD5ZGIBIKDKXDBEW3ZDP2PJ75PDMHUY2W6RQ, seg_pos={0 0}, upload=470152]: Uploading piece... Dec 27 23:33:49.384183156 (536h4m49.255266221s) [noise=true , address=145.79.227.38:4978 , piece_id=A4Y464TSTACJJHH3OOLFPFT4TJARXUMNSWDMTNZQCELAGS7KS6PA, seg_pos={0 0}, upload=470152]: Uploading piece... Dec 27 23:33:49.385074854 (536h4m49.256157899s) [noise=true , address=145.79.227.16:4910 , piece_id=UGT7TNS5IYSGENTUJQLISILCAAFHREIWCA2BQRCEQNDRSSKSVGJQ, seg_pos={0 0}, upload=470152]: Uploading piece... Dec 27 23:33:51.077968573 (536h4m50.949051618s) [upload=470152]: Next returned nil segment Dec 27 23:33:51.077984935 (536h4m50.94906797s) [upload=470152]: Waiting for error group managing segments... Dec 27 23:33:51.364858338 (536h4m51.235941384s) [noise=true , address=145.79.227.32:4836 , piece_id=HDI3NBNAB3WGFK6DPZVG74UMIBHOYWKWOSPYFLLBL34NHYLMJPLQ, seg_pos={0 0}, upload=470152]: Done uploading piece. err: <nil> Dec 27 23:33:51.365609008 (536h4m51.236692053s) [noise=true , address=145.79.227.12:4765 , piece_id=7C2KD5PG35IEYDYLJ2RBYAAAR6PGHUFMPB6Y7BF4ND4ISRAR67AQ, seg_pos={0 0}, upload=470152]: Done uploading piece. err: <nil> Dec 27 23:33:51.386380872 (536h4m51.257463907s) [noise=true , address=145.79.227.38:4978 , piece_id=A4Y464TSTACJJHH3OOLFPFT4TJARXUMNSWDMTNZQCELAGS7KS6PA, seg_pos={0 0}, upload=470152]: Done uploading piece. err: <nil> Dec 27 23:33:51.393609195 (536h4m51.26469224s) [noise=true , address=145.79.227.2:4113 , piece_id=75KW7BRUFS57ZPJQHA6L4F7NHI5OZCBYUACMT46SC6SMACQT3NHQ, seg_pos={0 0}, upload=470152]: Done uploading piece. err: <nil> Dec 27 23:33:51.397245198 (536h4m51.268328233s) [noise=true , address=145.79.227.11:4729 , piece_id=5ZKLEDW2ACLXWU5BNOKZYPTJWTNH4NEUEFURKF6KP4NTBKHZ2GRQ, seg_pos={0 0}, upload=470152]: Done uploading piece. err: <nil> Dec 27 23:33:51.402429069 (536h4m51.273512114s) [noise=true , address=145.79.227.42:4121 , piece_id=X5TLQLAGJTNVX7QLTZZEJQ5DIXH24LDOFDFTCR5TULFV4NNAUXXA, seg_pos={0 0}, upload=470152]: Done uploading piece. err: <nil> Dec 27 23:33:51.404353073 (536h4m51.275436118s) [noise=true , address=145.79.227.9:4365 , piece_id=H3MS6PAY6OSTMZ53ALUKWWPES52W3BVZ2AA6KBPW3XR5EQTAVFDA, seg_pos={0 0}, upload=470152]: Done uploading piece. err: <nil> Dec 27 23:33:51.465026535 (536h4m51.336109571s) [noise=true , address=145.79.227.16:4910 , piece_id=UGT7TNS5IYSGENTUJQLISILCAAFHREIWCA2BQRCEQNDRSSKSVGJQ, seg_pos={0 0}, upload=470152]: Done uploading piece. err: <nil> Dec 27 23:33:51.598230477 (536h4m51.469313522s) [noise=true , address=45.91.116.12:4228 , piece_id=B6IB7VXP5Q34RKRWRJNTRAPPOYFXD25VB4OVUSUHL7W3PQXSM7CA, seg_pos={0 0}, upload=470152]: Done uploading piece. err: <nil> Dec 27 23:33:51.620746652 (536h4m51.491829687s) [noise=true , address=77.90.31.12:4589 , piece_id=FAVB2HA4K346XXAMONOCIDT7IAWRM7PCXJXPKEYNSIGYW7YQEVLA, seg_pos={0 0}, upload=470152]: Done uploading piece. err: <nil> Dec 27 23:33:51.624783689 (536h4m51.495866734s) [noise=true , address=45.91.116.13:4153 , piece_id=DYZ7OQXT4BO2QX3PIBK2NMN3LCKDG5WJOTR5DUH7ZWHBEXY3Y6CA, seg_pos={0 0}, upload=470152]: Done uploading piece. err: <nil> Dec 27 23:33:51.691837321 (536h4m51.562920366s) [noise=true , address=145.79.227.48:4303 , piece_id=6UHP65PGTKISETS3VKRM472HFIKW2YWNHK7LQFG6RKDZDVCTST2A, seg_pos={0 0}, upload=470152]: Done uploading piece. err: <nil> Dec 27 23:33:51.696089648 (536h4m51.567172693s) [noise=true , address=45.91.116.4:4940 , piece_id=M55UKXPZ2XB2MEW7B5ZSWW4GT7FEHYWK2NJXSV3IA6H5JZJVPWTQ, seg_pos={0 0}, upload=470152]: Done uploading piece. err: <nil> Dec 27 23:33:51.696103364 (536h4m51.567186399s) [seg_pos={0 0}, upload=470152]: Segment reached optimal threshold of 13 pieces. Dec 27 23:33:51.696296061 (536h4m51.567379096s) [noise=true , address=145.79.227.40:4049 , piece_id=NI2W7IRRO4R4VZ3PDTFY4KMQA7LGVVQT42DPWSB3SC2KOTBCEPKA, seg_pos={0 0}, upload=470152]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 27 23:33:51.696303094 (536h4m51.567386129s) [noise=true , address=145.79.227.40:4049 , piece_id=NI2W7IRRO4R4VZ3PDTFY4KMQA7LGVVQT42DPWSB3SC2KOTBCEPKA, seg_pos={0 0}, upload=470152]: Long tail cancellation due to optimal threshold Dec 27 23:33:51.696419936 (536h4m51.567502971s) [noise=true , address=145.79.227.37:4567 , piece_id=PNNETAJYZ7PY6GL2WD5ZGIBIKDKXDBEW3ZDP2PJ75PDMHUY2W6RQ, seg_pos={0 0}, upload=470152]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:46664->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:46664->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46664->145.79.227.37:4567: use of closed network connection Dec 27 23:33:51.696425447 (536h4m51.567508482s) [noise=true , address=145.79.227.37:4567 , piece_id=PNNETAJYZ7PY6GL2WD5ZGIBIKDKXDBEW3ZDP2PJ75PDMHUY2W6RQ, seg_pos={0 0}, upload=470152]: Long tail cancellation due to optimal threshold Dec 27 23:33:51.696742230 (536h4m51.567825275s) [noise=true , address=45.91.116.11:4117 , piece_id=QQCPS2JHRSGK4HPU5XA43WOQILXNVOQOL4YIZCKXBASWZJV6I4JA, seg_pos={0 0}, upload=470152]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 27 23:33:51.696760124 (536h4m51.567843169s) [noise=true , address=45.91.116.11:4117 , piece_id=QQCPS2JHRSGK4HPU5XA43WOQILXNVOQOL4YIZCKXBASWZJV6I4JA, seg_pos={0 0}, upload=470152]: Long tail cancellation due to optimal threshold Dec 27 23:33:51.696806723 (536h4m51.567889758s) [seg_pos={0 0}, upload=470152]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:33:51.696894540 (536h4m51.567977585s) [upload=470152]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:33:51.696902105 (536h4m51.56798515s) [upload=470152]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:33:52.008814236 (536h4m51.879897291s) [upload=470152]: Done uploading segments... Dec 27 23:33:52.008836979 (536h4m51.879920014s) [upload=470152]: Upload finished. err: <nil> Duration: 30.003565723s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....X.....X..........................XXX.....XX.XX.XXXX.XXXX] duration: 30.003191907s trace: https://jaeger.dev.storj.io/trace/741c1f0a8e3babb6 FAIL (2025-12-27 23:33:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: L2TIQQNRSCACZYQ4AADYJTOYVTZYDRQQKJBWA5TKBHG6LKXW643A) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: HS5AKIDQUG46RFTRRNZF2REMW5DPSSDGJA6XJEILPXOVAT2DVYYQ) context deadline exceeded ======LOGS====== Dec 27 23:33:03.391044913 (536h4m3.262127958s) [upload=470141]: Starting upload Dec 27 23:33:03.391223102 (536h4m3.262306147s) [upload=470141]: Done starting upload Dec 27 23:33:03.391337861 (536h4m3.262420906s) [upload=470141]: Uploading segments... Dec 27 23:33:03.391573459 (536h4m3.262656514s) [upload=470141]: Got next segment. Inline: false Dec 27 23:33:03.391702785 (536h4m3.26278582s) [upload=470141]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:33:03.391710049 (536h4m3.262793094s) [upload=470141]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:33:03.632699092 (536h4m3.503782147s) [seg_pos={0 0}, upload=470141]: Begin upload segment... Dec 27 23:33:03.632753135 (536h4m3.50383617s) [seg_pos={0 0}, upload=470141]: Done begin upload segment. Dec 27 23:33:03.632786699 (536h4m3.503869744s) [noise=true , address=145.79.227.3:4150 , piece_id=OP5DXORYCTZA5DXBEIFQMOSW6XREN4SPFBAET5XZMANEIHQPIEQQ, seg_pos={0 0}, upload=470141]: Uploading piece... Dec 27 23:33:03.632836213 (536h4m3.503919268s) [noise=true , address=45.91.116.8:4045 , piece_id=HS5AKIDQUG46RFTRRNZF2REMW5DPSSDGJA6XJEILPXOVAT2DVYYQ, seg_pos={0 0}, upload=470141]: Uploading piece... Dec 27 23:33:03.634230478 (536h4m3.505313523s) [noise=true , address=145.79.227.16:4911 , piece_id=7LBZZY4RJXJYPTUMMX4BNYSC22PJONFWMXXD2TJEHNV5WHAZWY7A, seg_pos={0 0}, upload=470141]: Uploading piece... Dec 27 23:33:03.634259093 (536h4m3.505342138s) [noise=true , address=145.79.227.37:4567 , piece_id=6GTI3ZAOEQBOFTMOAUYVEUHHEJ4Q2YSOUPHYRPY4EZM5ROHA3UPA, seg_pos={0 0}, upload=470141]: Uploading piece... Dec 27 23:33:03.634739598 (536h4m3.505822643s) [noise=true , address=77.90.31.19:4844 , piece_id=S4LL6RC3A5VBQ75FSRJNCMYPY3BJ4IDH52D6ITDLWLVHUIR4MGLA, seg_pos={0 0}, upload=470141]: Uploading piece... Dec 27 23:33:03.635241213 (536h4m3.506324258s) [noise=true , address=77.90.31.13:4627 , piece_id=73UBITPCIQLZZXPKIA22KTQNGVJ2GMFS3AUCLBQY7UHWO52XBY4A, seg_pos={0 0}, upload=470141]: Uploading piece... Dec 27 23:33:03.635278333 (536h4m3.506361388s) [noise=true , address=145.79.227.42:4121 , piece_id=6PBBEDAMBX6EU63LNDVD2QGEXYKQ5OM5F4V3BXW557G53REZVQQQ, seg_pos={0 0}, upload=470141]: Uploading piece... Dec 27 23:33:03.635733780 (536h4m3.506816825s) [noise=true , address=145.79.227.41:4087 , piece_id=JOTFFWPHRQEGG7VHTKNGOFCHJVTKEIBVJBRPATTPUPQPYLJHZ4OQ, seg_pos={0 0}, upload=470141]: Uploading piece... Dec 27 23:33:03.636036768 (536h4m3.507119813s) [noise=true , address=145.79.227.44:4195 , piece_id=ZGBYFLSLZ5DQSHUB33YE2TQWZYGZLPIRLDNE3VALSMDQ47PJCJWA, seg_pos={0 0}, upload=470141]: Uploading piece... Dec 27 23:33:03.636188486 (536h4m3.507271531s) [noise=true , address=145.79.227.19:4079 , piece_id=47B2HN2ROF3N6FTCCIQC7GZ4IVC7VANFTVRHCWRPNEVIWIKF7HKA, seg_pos={0 0}, upload=470141]: Uploading piece... Dec 27 23:33:03.636636289 (536h4m3.507719334s) [noise=true , address=145.79.227.11:4731 , piece_id=WITH7WYZMSGBUB7KR7EIZUGSB3CAX2KS4JDUOVU3GKVY6RI3BP5A, seg_pos={0 0}, upload=470141]: Uploading piece... Dec 27 23:33:03.636889191 (536h4m3.507972236s) [noise=true , address=145.79.227.31:4798 , piece_id=UXLSB2P7O3UZKGRAZF42SSB4NL6G4VICAERJKOGLG3K7B4Y5U7GQ, seg_pos={0 0}, upload=470141]: Uploading piece... Dec 27 23:33:03.637172811 (536h4m3.508255856s) [noise=true , address=77.90.31.15:4697 , piece_id=L2TIQQNRSCACZYQ4AADYJTOYVTZYDRQQKJBWA5TKBHG6LKXW643A, seg_pos={0 0}, upload=470141]: Uploading piece... Dec 27 23:33:03.637525262 (536h4m3.508608307s) [noise=true , address=145.79.227.51:4411 , piece_id=O4RI4SPIZONDM3EEVTFZK4GS6TNAY3Z2VSDEKXE5LDHZWVC6FTIA, seg_pos={0 0}, upload=470141]: Uploading piece... Dec 27 23:33:03.638390009 (536h4m3.509473054s) [noise=true , address=145.79.227.32:4836 , piece_id=5SIQOKUPEZBR7W4ZP7ZOLQS4VVT7AHYH4LRE6OVQVLW6452H3WFQ, seg_pos={0 0}, upload=470141]: Uploading piece... Dec 27 23:33:03.640071851 (536h4m3.511154896s) [noise=true , address=145.79.227.4:4186 , piece_id=PVNZ5CTPEGU7NQTLLFNTLT5UIBCJDRAOFIAZYXVYFJXRITMLPFGQ, seg_pos={0 0}, upload=470141]: Uploading piece... Dec 27 23:33:05.262214318 (536h4m5.133297353s) [upload=470141]: Next returned nil segment Dec 27 23:33:05.262241580 (536h4m5.133324615s) [upload=470141]: Waiting for error group managing segments... Dec 27 23:33:05.637044697 (536h4m5.508127732s) [noise=true , address=145.79.227.37:4567 , piece_id=6GTI3ZAOEQBOFTMOAUYVEUHHEJ4Q2YSOUPHYRPY4EZM5ROHA3UPA, seg_pos={0 0}, upload=470141]: Done uploading piece. err: <nil> Dec 27 23:33:05.640207549 (536h4m5.511290594s) [noise=true , address=145.79.227.42:4121 , piece_id=6PBBEDAMBX6EU63LNDVD2QGEXYKQ5OM5F4V3BXW557G53REZVQQQ, seg_pos={0 0}, upload=470141]: Done uploading piece. err: <nil> Dec 27 23:33:05.649735202 (536h4m5.520818277s) [noise=true , address=145.79.227.3:4150 , piece_id=OP5DXORYCTZA5DXBEIFQMOSW6XREN4SPFBAET5XZMANEIHQPIEQQ, seg_pos={0 0}, upload=470141]: Done uploading piece. err: <nil> Dec 27 23:33:05.661874416 (536h4m5.532957452s) [noise=true , address=145.79.227.4:4186 , piece_id=PVNZ5CTPEGU7NQTLLFNTLT5UIBCJDRAOFIAZYXVYFJXRITMLPFGQ, seg_pos={0 0}, upload=470141]: Done uploading piece. err: <nil> Dec 27 23:33:05.663052931 (536h4m5.534135976s) [noise=true , address=145.79.227.16:4911 , piece_id=7LBZZY4RJXJYPTUMMX4BNYSC22PJONFWMXXD2TJEHNV5WHAZWY7A, seg_pos={0 0}, upload=470141]: Done uploading piece. err: <nil> Dec 27 23:33:05.665082866 (536h4m5.536165901s) [noise=true , address=145.79.227.41:4087 , piece_id=JOTFFWPHRQEGG7VHTKNGOFCHJVTKEIBVJBRPATTPUPQPYLJHZ4OQ, seg_pos={0 0}, upload=470141]: Done uploading piece. err: <nil> Dec 27 23:33:05.666561772 (536h4m5.537644817s) [noise=true , address=145.79.227.32:4836 , piece_id=5SIQOKUPEZBR7W4ZP7ZOLQS4VVT7AHYH4LRE6OVQVLW6452H3WFQ, seg_pos={0 0}, upload=470141]: Done uploading piece. err: <nil> Dec 27 23:33:05.868161792 (536h4m5.739244827s) [noise=true , address=77.90.31.19:4844 , piece_id=S4LL6RC3A5VBQ75FSRJNCMYPY3BJ4IDH52D6ITDLWLVHUIR4MGLA, seg_pos={0 0}, upload=470141]: Done uploading piece. err: <nil> Dec 27 23:33:05.889053864 (536h4m5.760136919s) [noise=true , address=77.90.31.13:4627 , piece_id=73UBITPCIQLZZXPKIA22KTQNGVJ2GMFS3AUCLBQY7UHWO52XBY4A, seg_pos={0 0}, upload=470141]: Done uploading piece. err: <nil> Dec 27 23:33:05.896257361 (536h4m5.767340406s) [noise=true , address=77.90.31.15:4697 , piece_id=L2TIQQNRSCACZYQ4AADYJTOYVTZYDRQQKJBWA5TKBHG6LKXW643A, seg_pos={0 0}, upload=470141]: Done uploading piece. err: <nil> Dec 27 23:33:06.238915808 (536h4m6.109998843s) [noise=true , address=45.91.116.8:4045 , piece_id=HS5AKIDQUG46RFTRRNZF2REMW5DPSSDGJA6XJEILPXOVAT2DVYYQ, seg_pos={0 0}, upload=470141]: Done uploading piece. err: <nil> Dec 27 23:33:06.548953542 (536h4m6.420036577s) [noise=true , address=145.79.227.11:4731 , piece_id=WITH7WYZMSGBUB7KR7EIZUGSB3CAX2KS4JDUOVU3GKVY6RI3BP5A, seg_pos={0 0}, upload=470141]: Done uploading piece. err: <nil> Dec 27 23:33:07.502088318 (536h4m7.373171353s) [noise=true , address=145.79.227.44:4195 , piece_id=ZGBYFLSLZ5DQSHUB33YE2TQWZYGZLPIRLDNE3VALSMDQ47PJCJWA, seg_pos={0 0}, upload=470141]: Done uploading piece. err: <nil> Dec 27 23:33:07.502113797 (536h4m7.373196842s) [seg_pos={0 0}, upload=470141]: Segment reached optimal threshold of 13 pieces. Dec 27 23:33:07.502410782 (536h4m7.373493817s) [noise=true , address=145.79.227.19:4079 , piece_id=47B2HN2ROF3N6FTCCIQC7GZ4IVC7VANFTVRHCWRPNEVIWIKF7HKA, seg_pos={0 0}, upload=470141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: expected piece hash; context canceled; EOF Dec 27 23:33:07.502420952 (536h4m7.373503987s) [noise=true , address=145.79.227.19:4079 , piece_id=47B2HN2ROF3N6FTCCIQC7GZ4IVC7VANFTVRHCWRPNEVIWIKF7HKA, seg_pos={0 0}, upload=470141]: Long tail cancellation due to optimal threshold Dec 27 23:33:07.502651921 (536h4m7.373734966s) [noise=true , address=145.79.227.31:4798 , piece_id=UXLSB2P7O3UZKGRAZF42SSB4NL6G4VICAERJKOGLG3K7B4Y5U7GQ, seg_pos={0 0}, upload=470141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ): protocol: expected piece hash; context canceled; EOF Dec 27 23:33:07.502673623 (536h4m7.373756678s) [noise=true , address=145.79.227.31:4798 , piece_id=UXLSB2P7O3UZKGRAZF42SSB4NL6G4VICAERJKOGLG3K7B4Y5U7GQ, seg_pos={0 0}, upload=470141]: Long tail cancellation due to optimal threshold Dec 27 23:33:07.502755348 (536h4m7.373838393s) [noise=true , address=145.79.227.51:4411 , piece_id=O4RI4SPIZONDM3EEVTFZK4GS6TNAY3Z2VSDEKXE5LDHZWVC6FTIA, seg_pos={0 0}, upload=470141]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR): protocol: write tcp 188.166.225.170:35902->145.79.227.51:4411: use of closed network connection; write tcp 188.166.225.170:35902->145.79.227.51:4411: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35902->145.79.227.51:4411: use of closed network connection Dec 27 23:33:07.502765647 (536h4m7.373848702s) [noise=true , address=145.79.227.51:4411 , piece_id=O4RI4SPIZONDM3EEVTFZK4GS6TNAY3Z2VSDEKXE5LDHZWVC6FTIA, seg_pos={0 0}, upload=470141]: Long tail cancellation due to optimal threshold Dec 27 23:33:07.502874355 (536h4m7.37395739s) [seg_pos={0 0}, upload=470141]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:33:07.503056201 (536h4m7.374139236s) [upload=470141]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:33:07.503064577 (536h4m7.374147612s) [upload=470141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:33:07.794546667 (536h4m7.665629712s) [upload=470141]: Done uploading segments... Dec 27 23:33:07.794572356 (536h4m7.665655391s) [upload=470141]: Upload finished. err: <nil> Duration: 30.003224098s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.533457486s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:33:18 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.533482153s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.499263285s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:32:22 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.499290146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.....X.....X..........................XXX.....XX.XX.XXXX.XXX] duration: 30.00275098s trace: https://jaeger.dev.storj.io/trace/7875b3f1ae494511 FAIL (2025-12-27 23:32:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: G4VU5OENXLXESW2UTK6LPRXBTQ2V7EQ2BNJBFYSYU7CMPB3CUE2A) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: PCUHR6VFNIMJQV555XVJM7H7KRG4PLMPU47UUCPLTIJAA5QNCZFA) context deadline exceeded ======LOGS====== .431056218s) [upload=470123]: Uploading segments... Dec 27 23:31:49.560056241 (536h2m49.431139286s) [upload=470123]: Got next segment. Inline: false Dec 27 23:31:49.560105554 (536h2m49.431188599s) [upload=470123]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:31:49.560116055 (536h2m49.43119913s) [upload=470123]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:31:49.804161949 (536h2m49.675244994s) [seg_pos={0 0}, upload=470123]: Begin upload segment... Dec 27 23:31:49.804220680 (536h2m49.675303725s) [seg_pos={0 0}, upload=470123]: Done begin upload segment. Dec 27 23:31:49.804254575 (536h2m49.67533761s) [noise=true , address=45.91.116.6:4976 , piece_id=PCUHR6VFNIMJQV555XVJM7H7KRG4PLMPU47UUCPLTIJAA5QNCZFA, seg_pos={0 0}, upload=470123]: Uploading piece... Dec 27 23:31:49.804268161 (536h2m49.675351196s) [noise=true , address=145.79.227.3:4150 , piece_id=FNXBIF7QXO6RVKZZNPVQHMNOVMU7VKXWIWFEPJAOX4RRKQWMK37A, seg_pos={0 0}, upload=470123]: Uploading piece... Dec 27 23:31:49.804736823 (536h2m49.675819858s) [noise=true , address=45.91.116.8:4047 , piece_id=LFGRMUMMXBRI6Y2MEABEZ2JHD24KOPENQVT6YMQEFSGMQDTXBVCQ, seg_pos={0 0}, upload=470123]: Uploading piece... Dec 27 23:31:49.805162994 (536h2m49.676246029s) [noise=true , address=145.79.227.2:4115 , piece_id=PUGXZ22O7ZJJLLZOERCSZUVYETFDCJUNVH4LI472Y7TIUSQUC63Q, seg_pos={0 0}, upload=470123]: Uploading piece... Dec 27 23:31:49.805610567 (536h2m49.676693602s) [noise=true , address=145.79.227.15:4874 , piece_id=ZHMUOR7Y5UPDEM74MI33YC4YUUK5P2UNUXXAAI5PJTNY7NT5CGDQ, seg_pos={0 0}, upload=470123]: Uploading piece... Dec 27 23:31:49.806083257 (536h2m49.677166302s) [noise=true , address=145.79.227.50:4375 , piece_id=7R6KRGJ6MV5MJCI43PQSPMGNIGOF52K2NITX5GHAAN4ACIZZAGWQ, seg_pos={0 0}, upload=470123]: Uploading piece... Dec 27 23:31:49.806540257 (536h2m49.677623302s) [noise=true , address=145.79.227.7:4293 , piece_id=LIFA7BJ465M6PKAKF7P2QTUXXXSYGDHBGVPYXE2GXHMCVO2EMGMQ, seg_pos={0 0}, upload=470123]: Uploading piece... Dec 27 23:31:49.806982890 (536h2m49.678065936s) [noise=true , address=145.79.227.9:4365 , piece_id=PA7T6PTGF6WI34SDWCWJPBNJNYLEO4KCMI6VNDBTZWFSAKCUQT3A, seg_pos={0 0}, upload=470123]: Uploading piece... Dec 27 23:31:49.807920997 (536h2m49.679004032s) [noise=true , address=145.79.227.42:4121 , piece_id=OB7ZTKVWNDURKS3IEEN2H6VSNP7SHWCIWJZDPFZDGNYGKD5MTOLQ, seg_pos={0 0}, upload=470123]: Uploading piece... Dec 27 23:31:49.808632511 (536h2m49.679715556s) [noise=true , address=145.79.227.11:4731 , piece_id=P2O7HUMEK5E5Q2U25CXO4TBLC2J2E6JNLFNQ5GTLDYSKK3D372LA, seg_pos={0 0}, upload=470123]: Uploading piece... Dec 27 23:31:49.810391430 (536h2m49.681474475s) [noise=true , address=77.90.31.14:4663 , piece_id=GZFSXNOYG2XSZN7ODSW2PMJSA6O7N537A7ICNFW7IN2QOCDHBF3A, seg_pos={0 0}, upload=470123]: Uploading piece... Dec 27 23:31:49.811892559 (536h2m49.682975604s) [noise=true , address=145.79.227.21:4473 , piece_id=U4YOCKSRZ6MH23LCMNBI6SGMF7VIOW27LIENU73TLG5JSWVZEA6Q, seg_pos={0 0}, upload=470123]: Uploading piece... Dec 27 23:31:49.813506783 (536h2m49.684589828s) [noise=true , address=77.90.31.11:4554 , piece_id=4ZRKEYE2Q4E2KO4TBOI636BY5Q5HQJM4JVFTJ5IWPWZHFS65B4ZQ, seg_pos={0 0}, upload=470123]: Uploading piece... Dec 27 23:31:49.815170210 (536h2m49.686253245s) [noise=true , address=45.91.116.10:4084 , piece_id=G4VU5OENXLXESW2UTK6LPRXBTQ2V7EQ2BNJBFYSYU7CMPB3CUE2A, seg_pos={0 0}, upload=470123]: Uploading piece... Dec 27 23:31:49.816004349 (536h2m49.687087394s) [noise=true , address=145.79.227.4:4185 , piece_id=ESVZJT36VWV4DDSLFCN6IIEXOERHP7XD2BLJ4ZKTTH5MNCVIGUOQ, seg_pos={0 0}, upload=470123]: Uploading piece... Dec 27 23:31:49.816729580 (536h2m49.687812625s) [noise=true , address=45.91.116.9:4263 , piece_id=FBFXNX47OOIPQ4BWXD6AV6RXA3LD45ZTCF6LB7ONPCDGPWM3SZHQ, seg_pos={0 0}, upload=470123]: Uploading piece... Dec 27 23:31:51.428218120 (536h2m51.299301155s) [upload=470123]: Next returned nil segment Dec 27 23:31:51.428304234 (536h2m51.299387299s) [upload=470123]: Waiting for error group managing segments... Dec 27 23:31:51.810909057 (536h2m51.681992102s) [noise=true , address=145.79.227.42:4121 , piece_id=OB7ZTKVWNDURKS3IEEN2H6VSNP7SHWCIWJZDPFZDGNYGKD5MTOLQ, seg_pos={0 0}, upload=470123]: Done uploading piece. err: <nil> Dec 27 23:31:51.811963505 (536h2m51.68304654s) [noise=true , address=145.79.227.9:4365 , piece_id=PA7T6PTGF6WI34SDWCWJPBNJNYLEO4KCMI6VNDBTZWFSAKCUQT3A, seg_pos={0 0}, upload=470123]: Done uploading piece. err: <nil> Dec 27 23:31:51.815235826 (536h2m51.686318871s) [noise=true , address=145.79.227.15:4874 , piece_id=ZHMUOR7Y5UPDEM74MI33YC4YUUK5P2UNUXXAAI5PJTNY7NT5CGDQ, seg_pos={0 0}, upload=470123]: Done uploading piece. err: <nil> Dec 27 23:31:51.826625414 (536h2m51.697708449s) [noise=true , address=145.79.227.11:4731 , piece_id=P2O7HUMEK5E5Q2U25CXO4TBLC2J2E6JNLFNQ5GTLDYSKK3D372LA, seg_pos={0 0}, upload=470123]: Done uploading piece. err: <nil> Dec 27 23:31:51.829534483 (536h2m51.700617528s) [noise=true , address=145.79.227.21:4473 , piece_id=U4YOCKSRZ6MH23LCMNBI6SGMF7VIOW27LIENU73TLG5JSWVZEA6Q, seg_pos={0 0}, upload=470123]: Done uploading piece. err: <nil> Dec 27 23:31:51.833598872 (536h2m51.704681917s) [noise=true , address=145.79.227.3:4150 , piece_id=FNXBIF7QXO6RVKZZNPVQHMNOVMU7VKXWIWFEPJAOX4RRKQWMK37A, seg_pos={0 0}, upload=470123]: Done uploading piece. err: <nil> Dec 27 23:31:51.838062111 (536h2m51.709145146s) [noise=true , address=145.79.227.4:4185 , piece_id=ESVZJT36VWV4DDSLFCN6IIEXOERHP7XD2BLJ4ZKTTH5MNCVIGUOQ, seg_pos={0 0}, upload=470123]: Done uploading piece. err: <nil> Dec 27 23:31:51.846783548 (536h2m51.717866583s) [noise=true , address=145.79.227.7:4293 , piece_id=LIFA7BJ465M6PKAKF7P2QTUXXXSYGDHBGVPYXE2GXHMCVO2EMGMQ, seg_pos={0 0}, upload=470123]: Done uploading piece. err: <nil> Dec 27 23:31:52.040351243 (536h2m51.911434278s) [noise=true , address=45.91.116.8:4047 , piece_id=LFGRMUMMXBRI6Y2MEABEZ2JHD24KOPENQVT6YMQEFSGMQDTXBVCQ, seg_pos={0 0}, upload=470123]: Done uploading piece. err: <nil> Dec 27 23:31:52.060067616 (536h2m51.931150651s) [noise=true , address=77.90.31.11:4554 , piece_id=4ZRKEYE2Q4E2KO4TBOI636BY5Q5HQJM4JVFTJ5IWPWZHFS65B4ZQ, seg_pos={0 0}, upload=470123]: Done uploading piece. err: <nil> Dec 27 23:31:52.099325379 (536h2m51.970408424s) [noise=true , address=45.91.116.6:4976 , piece_id=PCUHR6VFNIMJQV555XVJM7H7KRG4PLMPU47UUCPLTIJAA5QNCZFA, seg_pos={0 0}, upload=470123]: Done uploading piece. err: <nil> Dec 27 23:31:52.119817630 (536h2m51.990900675s) [noise=true , address=45.91.116.9:4263 , piece_id=FBFXNX47OOIPQ4BWXD6AV6RXA3LD45ZTCF6LB7ONPCDGPWM3SZHQ, seg_pos={0 0}, upload=470123]: Done uploading piece. err: <nil> Dec 27 23:31:52.793062549 (536h2m52.664145594s) [noise=true , address=45.91.116.10:4084 , piece_id=G4VU5OENXLXESW2UTK6LPRXBTQ2V7EQ2BNJBFYSYU7CMPB3CUE2A, seg_pos={0 0}, upload=470123]: Done uploading piece. err: <nil> Dec 27 23:31:52.793085743 (536h2m52.664168778s) [seg_pos={0 0}, upload=470123]: Segment reached optimal threshold of 13 pieces. Dec 27 23:31:52.793274393 (536h2m52.664357428s) [noise=true , address=77.90.31.14:4663 , piece_id=GZFSXNOYG2XSZN7ODSW2PMJSA6O7N537A7ICNFW7IN2QOCDHBF3A, seg_pos={0 0}, upload=470123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 27 23:31:52.793282498 (536h2m52.664365563s) [noise=true , address=77.90.31.14:4663 , piece_id=GZFSXNOYG2XSZN7ODSW2PMJSA6O7N537A7ICNFW7IN2QOCDHBF3A, seg_pos={0 0}, upload=470123]: Long tail cancellation due to optimal threshold Dec 27 23:31:52.793314338 (536h2m52.664397374s) [noise=true , address=145.79.227.50:4375 , piece_id=7R6KRGJ6MV5MJCI43PQSPMGNIGOF52K2NITX5GHAAN4ACIZZAGWQ, seg_pos={0 0}, upload=470123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: write tcp 188.166.225.170:58064->145.79.227.50:4375: use of closed network connection; write tcp 188.166.225.170:58064->145.79.227.50:4375: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58064->145.79.227.50:4375: use of closed network connection Dec 27 23:31:52.793323766 (536h2m52.664406801s) [noise=true , address=145.79.227.50:4375 , piece_id=7R6KRGJ6MV5MJCI43PQSPMGNIGOF52K2NITX5GHAAN4ACIZZAGWQ, seg_pos={0 0}, upload=470123]: Long tail cancellation due to optimal threshold Dec 27 23:31:52.793487168 (536h2m52.664570203s) [noise=true , address=145.79.227.2:4115 , piece_id=PUGXZ22O7ZJJLLZOERCSZUVYETFDCJUNVH4LI472Y7TIUSQUC63Q, seg_pos={0 0}, upload=470123]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR): protocol: write tcp 188.166.225.170:59334->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:59334->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59334->145.79.227.2:4115: use of closed network connection Dec 27 23:31:52.793492758 (536h2m52.664575793s) [noise=true , address=145.79.227.2:4115 , piece_id=PUGXZ22O7ZJJLLZOERCSZUVYETFDCJUNVH4LI472Y7TIUSQUC63Q, seg_pos={0 0}, upload=470123]: Long tail cancellation due to optimal threshold Dec 27 23:31:52.793506855 (536h2m52.66458989s) [seg_pos={0 0}, upload=470123]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:31:52.793604000 (536h2m52.664687045s) [upload=470123]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:31:52.793612035 (536h2m52.66469508s) [upload=470123]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:31:53.082609133 (536h2m52.953692178s) [upload=470123]: Done uploading segments... Dec 27 23:31:53.082629222 (536h2m52.953712257s) [upload=470123]: Upload finished. err: <nil> Duration: 30.002773894s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [......X.....X..........................XXX.....XX.XX.XXXX.XX] duration: 30.003060326s trace: https://jaeger.dev.storj.io/trace/19c526d104dd0cbe FAIL (2025-12-27 23:31:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: J44B5256B3HQOQLUKBYTZTUUAYLXZLTYXXFBPCPGNWM3TMV6NTOA) context deadline exceeded; piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: IRUBUQMPGSEK56BPLA4TRNHFOJG474GX6D45JKBHE6WCSMXHYJRQ) context deadline exceeded ======LOGS====== Dec 27 23:31:02.242873380 (536h2m2.113956425s) [upload=470111]: Starting upload Dec 27 23:31:02.242975074 (536h2m2.114058119s) [upload=470111]: Done starting upload Dec 27 23:31:02.243213738 (536h2m2.114296774s) [upload=470111]: Uploading segments... Dec 27 23:31:02.243273692 (536h2m2.114356737s) [upload=470111]: Got next segment. Inline: false Dec 27 23:31:02.243304681 (536h2m2.114387726s) [upload=470111]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:31:02.243309240 (536h2m2.114392275s) [upload=470111]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:31:02.488886580 (536h2m2.359969625s) [seg_pos={0 0}, upload=470111]: Begin upload segment... Dec 27 23:31:02.488950592 (536h2m2.360033637s) [seg_pos={0 0}, upload=470111]: Done begin upload segment. Dec 27 23:31:02.489006629 (536h2m2.360089674s) [noise=true , address=45.91.116.13:4155 , piece_id=WJIPFOIB3UFHCSRB5V2AJ3IBC3WJG36MX2GVJLGYS5CARF6MDEHQ, seg_pos={0 0}, upload=470111]: Uploading piece... Dec 27 23:31:02.489089717 (536h2m2.360172762s) [noise=true , address=145.79.227.47:4267 , piece_id=YIAT5DNWKF7BJODBLTZTCJ2A5ITCV74SBFSTRB3FJ3PVDCSAXSEA, seg_pos={0 0}, upload=470111]: Uploading piece... Dec 27 23:31:02.489706081 (536h2m2.360789116s) [noise=true , address=77.90.31.13:4627 , piece_id=J44B5256B3HQOQLUKBYTZTUUAYLXZLTYXXFBPCPGNWM3TMV6NTOA, seg_pos={0 0}, upload=470111]: Uploading piece... Dec 27 23:31:02.490214177 (536h2m2.361297212s) [noise=true , address=45.91.116.7:4011 , piece_id=3N7R6UY6D2REBJGOTTODNC4RTD7S2YULRYGIIAR7TZFAKGMTALQA, seg_pos={0 0}, upload=470111]: Uploading piece... Dec 27 23:31:02.490688792 (536h2m2.361771837s) [noise=true , address=45.91.116.9:4262 , piece_id=IRUBUQMPGSEK56BPLA4TRNHFOJG474GX6D45JKBHE6WCSMXHYJRQ, seg_pos={0 0}, upload=470111]: Uploading piece... Dec 27 23:31:02.491127116 (536h2m2.362210161s) [noise=true , address=45.91.116.14:4191 , piece_id=R3ZLVXNDH5C2RBJN3AIH4UZA77JROZFHGMD7MHL3DZCN7K2YXBJA, seg_pos={0 0}, upload=470111]: Uploading piece... Dec 27 23:31:02.491560852 (536h2m2.362643897s) [noise=true , address=145.79.227.13:4804 , piece_id=DPFPAJUSV75LOFKSIDN5WN5RWO5EY36STZ327COHALHK3YMIZX4A, seg_pos={0 0}, upload=470111]: Uploading piece... Dec 27 23:31:02.492005228 (536h2m2.363088274s) [noise=true , address=145.79.227.9:4365 , piece_id=UARX4MICKGEVA6YEIXXY2ENE5NO27MLGHXQTOGY6IXAT5RPHKKLQ, seg_pos={0 0}, upload=470111]: Uploading piece... Dec 27 23:31:02.492522914 (536h2m2.363605959s) [noise=true , address=145.79.227.15:4873 , piece_id=EQG2YAMA55YDGRTRIVV2TRWW2WIWO47YPAHTYJORB63TWKL74YXQ, seg_pos={0 0}, upload=470111]: Uploading piece... Dec 27 23:31:02.494753462 (536h2m2.365836497s) [noise=true , address=145.79.227.52:4447 , piece_id=EMHJ4NAV3IIPNH45UNCMZXK2SK6A5N6VEARHOKYFPVWAFCKAYARA, seg_pos={0 0}, upload=470111]: Uploading piece... Dec 27 23:31:02.496481702 (536h2m2.367564747s) [noise=true , address=145.79.227.50:4374 , piece_id=WH5FJBHCQMM73XU6KPQMZ2BZA6FJ2ARY6JMDLIMQXKN66NG6CR6A, seg_pos={0 0}, upload=470111]: Uploading piece... Dec 27 23:31:02.498233288 (536h2m2.369316333s) [noise=true , address=145.79.227.31:4798 , piece_id=GFYHO52F3RQ6TTRRMFSPG75HBBD3AKQKHNQRLKO7MY5J3ZCTIUBA, seg_pos={0 0}, upload=470111]: Uploading piece... Dec 27 23:31:02.500479104 (536h2m2.371562149s) [noise=true , address=145.79.227.48:4303 , piece_id=SLR4ZXUPZMXUVSWJ7B3QFJ57XNTAMEOHPPCSWSQOACAEWYIJLFDQ, seg_pos={0 0}, upload=470111]: Uploading piece... Dec 27 23:31:02.502525210 (536h2m2.373608255s) [noise=true , address=45.91.116.5:4300 , piece_id=2LHU3BKNO2TR236Z2LXXCKW6AOV33UAY2BHPFJL4NBCQX7KH5CSA, seg_pos={0 0}, upload=470111]: Uploading piece... Dec 27 23:31:02.504224746 (536h2m2.375307792s) [noise=true , address=45.91.116.10:4081 , piece_id=MX4PS2SZA2DVOYR7XCPP6KGFZU6YG5KKEYIUWH3U4OW3PSVZVOYA, seg_pos={0 0}, upload=470111]: Uploading piece... Dec 27 23:31:02.505974448 (536h2m2.377057493s) [noise=true , address=145.79.227.51:4409 , piece_id=PJDI7XDE5WPRUIIXGLWDNMJBILSSS4G5QV26YZWSOFS556MDSALA, seg_pos={0 0}, upload=470111]: Uploading piece... Dec 27 23:31:04.112714337 (536h2m3.983797382s) [upload=470111]: Next returned nil segment Dec 27 23:31:04.112728143 (536h2m3.983811188s) [upload=470111]: Waiting for error group managing segments... Dec 27 23:31:04.490740381 (536h2m4.361823426s) [noise=true , address=145.79.227.9:4365 , piece_id=UARX4MICKGEVA6YEIXXY2ENE5NO27MLGHXQTOGY6IXAT5RPHKKLQ, seg_pos={0 0}, upload=470111]: Done uploading piece. err: <nil> Dec 27 23:31:04.500097759 (536h2m4.371180794s) [noise=true , address=145.79.227.15:4873 , piece_id=EQG2YAMA55YDGRTRIVV2TRWW2WIWO47YPAHTYJORB63TWKL74YXQ, seg_pos={0 0}, upload=470111]: Done uploading piece. err: <nil> Dec 27 23:31:04.510474437 (536h2m4.381557482s) [noise=true , address=145.79.227.50:4374 , piece_id=WH5FJBHCQMM73XU6KPQMZ2BZA6FJ2ARY6JMDLIMQXKN66NG6CR6A, seg_pos={0 0}, upload=470111]: Done uploading piece. err: <nil> Dec 27 23:31:04.515265891 (536h2m4.386348936s) [noise=true , address=145.79.227.51:4409 , piece_id=PJDI7XDE5WPRUIIXGLWDNMJBILSSS4G5QV26YZWSOFS556MDSALA, seg_pos={0 0}, upload=470111]: Done uploading piece. err: <nil> Dec 27 23:31:04.515847940 (536h2m4.386930995s) [noise=true , address=145.79.227.52:4447 , piece_id=EMHJ4NAV3IIPNH45UNCMZXK2SK6A5N6VEARHOKYFPVWAFCKAYARA, seg_pos={0 0}, upload=470111]: Done uploading piece. err: <nil> Dec 27 23:31:04.515930837 (536h2m4.387013882s) [noise=true , address=145.79.227.31:4798 , piece_id=GFYHO52F3RQ6TTRRMFSPG75HBBD3AKQKHNQRLKO7MY5J3ZCTIUBA, seg_pos={0 0}, upload=470111]: Done uploading piece. err: <nil> Dec 27 23:31:04.520597083 (536h2m4.391680118s) [noise=true , address=145.79.227.13:4804 , piece_id=DPFPAJUSV75LOFKSIDN5WN5RWO5EY36STZ327COHALHK3YMIZX4A, seg_pos={0 0}, upload=470111]: Done uploading piece. err: <nil> Dec 27 23:31:04.723163713 (536h2m4.594246759s) [noise=true , address=45.91.116.9:4262 , piece_id=IRUBUQMPGSEK56BPLA4TRNHFOJG474GX6D45JKBHE6WCSMXHYJRQ, seg_pos={0 0}, upload=470111]: Done uploading piece. err: <nil> Dec 27 23:31:04.739139043 (536h2m4.610222088s) [noise=true , address=45.91.116.13:4155 , piece_id=WJIPFOIB3UFHCSRB5V2AJ3IBC3WJG36MX2GVJLGYS5CARF6MDEHQ, seg_pos={0 0}, upload=470111]: Done uploading piece. err: <nil> Dec 27 23:31:04.750367934 (536h2m4.62145098s) [noise=true , address=45.91.116.7:4011 , piece_id=3N7R6UY6D2REBJGOTTODNC4RTD7S2YULRYGIIAR7TZFAKGMTALQA, seg_pos={0 0}, upload=470111]: Done uploading piece. err: <nil> Dec 27 23:31:04.753778308 (536h2m4.624861354s) [noise=true , address=77.90.31.13:4627 , piece_id=J44B5256B3HQOQLUKBYTZTUUAYLXZLTYXXFBPCPGNWM3TMV6NTOA, seg_pos={0 0}, upload=470111]: Done uploading piece. err: <nil> Dec 27 23:31:04.805047934 (536h2m4.676130969s) [noise=true , address=45.91.116.14:4191 , piece_id=R3ZLVXNDH5C2RBJN3AIH4UZA77JROZFHGMD7MHL3DZCN7K2YXBJA, seg_pos={0 0}, upload=470111]: Done uploading piece. err: <nil> Dec 27 23:31:04.834553568 (536h2m4.705636603s) [noise=true , address=45.91.116.5:4300 , piece_id=2LHU3BKNO2TR236Z2LXXCKW6AOV33UAY2BHPFJL4NBCQX7KH5CSA, seg_pos={0 0}, upload=470111]: Done uploading piece. err: <nil> Dec 27 23:31:04.834572584 (536h2m4.705655619s) [seg_pos={0 0}, upload=470111]: Segment reached optimal threshold of 13 pieces. Dec 27 23:31:04.834773386 (536h2m4.705856431s) [noise=true , address=145.79.227.47:4267 , piece_id=YIAT5DNWKF7BJODBLTZTCJ2A5ITCV74SBFSTRB3FJ3PVDCSAXSEA, seg_pos={0 0}, upload=470111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: expected piece hash; context canceled; EOF Dec 27 23:31:04.834779628 (536h2m4.705862663s) [noise=true , address=145.79.227.47:4267 , piece_id=YIAT5DNWKF7BJODBLTZTCJ2A5ITCV74SBFSTRB3FJ3PVDCSAXSEA, seg_pos={0 0}, upload=470111]: Long tail cancellation due to optimal threshold Dec 27 23:31:04.835037860 (536h2m4.706120915s) [noise=true , address=145.79.227.48:4303 , piece_id=SLR4ZXUPZMXUVSWJ7B3QFJ57XNTAMEOHPPCSWSQOACAEWYIJLFDQ, seg_pos={0 0}, upload=470111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: write tcp 188.166.225.170:46128->145.79.227.48:4303: use of closed network connection; write tcp 188.166.225.170:46128->145.79.227.48:4303: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46128->145.79.227.48:4303: use of closed network connection Dec 27 23:31:04.835047288 (536h2m4.706130323s) [noise=true , address=145.79.227.48:4303 , piece_id=SLR4ZXUPZMXUVSWJ7B3QFJ57XNTAMEOHPPCSWSQOACAEWYIJLFDQ, seg_pos={0 0}, upload=470111]: Long tail cancellation due to optimal threshold Dec 27 23:31:04.835136067 (536h2m4.706219112s) [noise=true , address=45.91.116.10:4081 , piece_id=MX4PS2SZA2DVOYR7XCPP6KGFZU6YG5KKEYIUWH3U4OW3PSVZVOYA, seg_pos={0 0}, upload=470111]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 27 23:31:04.835145635 (536h2m4.706228671s) [noise=true , address=45.91.116.10:4081 , piece_id=MX4PS2SZA2DVOYR7XCPP6KGFZU6YG5KKEYIUWH3U4OW3PSVZVOYA, seg_pos={0 0}, upload=470111]: Long tail cancellation due to optimal threshold Dec 27 23:31:04.835167767 (536h2m4.706250812s) [seg_pos={0 0}, upload=470111]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:31:04.835288738 (536h2m4.706371773s) [upload=470111]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:31:04.835294288 (536h2m4.706377333s) [upload=470111]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:31:05.135042278 (536h2m5.006125323s) [upload=470111]: Done uploading segments... Dec 27 23:31:05.135069180 (536h2m5.006152235s) [upload=470111]: Upload finished. err: <nil> Duration: 30.003086997s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.58296342s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:31:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.582990231s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.574607162s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:30:26 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.574631999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......X.....X..........................XXX.....XX.XX.XXXX.X] duration: 30.002133145s trace: https://jaeger.dev.storj.io/trace/1904bc3b8627b9b6 FAIL (2025-12-27 23:30:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: XM3UALWAHCDL6DMO5ZGKE7WG2J4YXINC4DMDFGKPL5BRFGYWI63A) context deadline exceeded; piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: GDBOPGPGGZWM4UMG3SSV3PL2IU444Q2CR75RUMYZNFZAQXG3YWCA) context deadline exceeded ======LOGS====== 533608217s) [upload=470096]: Uploading segments... Dec 27 23:29:53.662693501 (536h0m53.533776576s) [upload=470096]: Got next segment. Inline: false Dec 27 23:29:53.662753676 (536h0m53.533836711s) [upload=470096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:29:53.662760970 (536h0m53.533844065s) [upload=470096]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:29:53.918512162 (536h0m53.789595197s) [seg_pos={0 0}, upload=470096]: Begin upload segment... Dec 27 23:29:53.918559682 (536h0m53.789642717s) [seg_pos={0 0}, upload=470096]: Done begin upload segment. Dec 27 23:29:53.918602725 (536h0m53.78968576s) [noise=true , address=145.79.227.36:4518 , piece_id=2V2XOJPMKI4W45JXL2IM2VTH5N5BLMBJK3HLTMXNXNWBAZUOHPJA, seg_pos={0 0}, upload=470096]: Uploading piece... Dec 27 23:29:53.921882520 (536h0m53.792965555s) [noise=true , address=145.79.227.15:4876 , piece_id=GUCTGD6Q5FLPIDQGUA7AFARRRDXPSQGGDKVKRP6ZM6SXZG4LLY2A, seg_pos={0 0}, upload=470096]: Uploading piece... Dec 27 23:29:53.924030571 (536h0m53.795113606s) [noise=true , address=77.90.31.14:4664 , piece_id=GDBOPGPGGZWM4UMG3SSV3PL2IU444Q2CR75RUMYZNFZAQXG3YWCA, seg_pos={0 0}, upload=470096]: Uploading piece... Dec 27 23:29:53.925187513 (536h0m53.796270549s) [noise=true , address=45.91.116.13:4156 , piece_id=C5WTVQ772CLEMKG627CU5OTO3A2SXZOQREI6ZOHFHVVPQHWNV7GQ, seg_pos={0 0}, upload=470096]: Uploading piece... Dec 27 23:29:53.926436572 (536h0m53.797519607s) [noise=true , address=77.90.31.15:4697 , piece_id=ORFKLJSAAG3F2CVTJASOHABK3R74FUTQZDIVOT3FSSSX4Z6Z3KFA, seg_pos={0 0}, upload=470096]: Uploading piece... Dec 27 23:29:53.926826184 (536h0m53.797909219s) [noise=true , address=145.79.227.31:4798 , piece_id=QGAJGTG33TYFU5KWXYA74GUYLIO2NWGAHCFAZJUU3GMUH6YARBTA, seg_pos={0 0}, upload=470096]: Uploading piece... Dec 27 23:29:53.927828963 (536h0m53.798912008s) [noise=true , address=145.79.227.16:4910 , piece_id=HSBDEVHKA7FEMFH2LETPO5PYJIM45RUVPXF4DDNE7WME5BHICWJA, seg_pos={0 0}, upload=470096]: Uploading piece... Dec 27 23:29:53.928374131 (536h0m53.799457166s) [noise=true , address=45.91.116.7:4010 , piece_id=RDEWAYTVTOMZ6U2PG4GUGTYFSBBMCTA6BW2I5CMHLIL7Z7ED5PLQ, seg_pos={0 0}, upload=470096]: Uploading piece... Dec 27 23:29:53.928910623 (536h0m53.799993658s) [noise=true , address=145.79.227.21:4473 , piece_id=LVJPMVISE4VGQYCF6ZGS5MEISPBR5NRDJYF6MSMVOM2LWWUEMQUA, seg_pos={0 0}, upload=470096]: Uploading piece... Dec 27 23:29:53.929837658 (536h0m53.800920693s) [noise=true , address=145.79.227.2:4114 , piece_id=JS7UXBFVLPCKSHVZFU2O3NLCMRMPGQUFXLKESLHWS2VCZ4NIWSLA, seg_pos={0 0}, upload=470096]: Uploading piece... Dec 27 23:29:53.930491333 (536h0m53.801574368s) [noise=true , address=145.79.227.32:4836 , piece_id=DXS4A2BKZZFTBZTBKVIY2WOOA3YVHGXE5SF3YPC2ITXD6RUCWZ6A, seg_pos={0 0}, upload=470096]: Uploading piece... Dec 27 23:29:53.931052591 (536h0m53.802135626s) [noise=true , address=45.91.116.14:4191 , piece_id=XM3UALWAHCDL6DMO5ZGKE7WG2J4YXINC4DMDFGKPL5BRFGYWI63A, seg_pos={0 0}, upload=470096]: Uploading piece... Dec 27 23:29:53.931994354 (536h0m53.803077389s) [noise=true , address=145.79.227.37:4567 , piece_id=ZHE7XUOKYHHVPQLW7U3E2BJLTSHGYNXAV65SUC4RIPM4266IWGVA, seg_pos={0 0}, upload=470096]: Uploading piece... Dec 27 23:29:53.932158587 (536h0m53.803241622s) [noise=true , address=145.79.227.13:4803 , piece_id=37N7HFRZ5AS3UVBJEZ2BFRZGZUY6DMTMWDQMXZB5FBUK5HEEACHA, seg_pos={0 0}, upload=470096]: Uploading piece... Dec 27 23:29:53.933317905 (536h0m53.80440094s) [noise=true , address=145.79.227.45:4229 , piece_id=XNV6OZ3OUDHAUJHOQZTTIPJP3OHBXWE7IPY2SSL2UR47UPOA24IA, seg_pos={0 0}, upload=470096]: Uploading piece... Dec 27 23:29:53.933360887 (536h0m53.804443932s) [noise=true , address=145.79.227.5:4221 , piece_id=FCU4S2AQPVA6XNWIYR7HN4LB6G5ZNALNO75ZOPSEHOQ5GSJLQGZQ, seg_pos={0 0}, upload=470096]: Uploading piece... Dec 27 23:29:55.640470224 (536h0m55.511553269s) [upload=470096]: Next returned nil segment Dec 27 23:29:55.640487187 (536h0m55.511570232s) [upload=470096]: Waiting for error group managing segments... Dec 27 23:29:55.920053587 (536h0m55.791136622s) [noise=true , address=145.79.227.45:4229 , piece_id=XNV6OZ3OUDHAUJHOQZTTIPJP3OHBXWE7IPY2SSL2UR47UPOA24IA, seg_pos={0 0}, upload=470096]: Done uploading piece. err: <nil> Dec 27 23:29:55.923379821 (536h0m55.794462856s) [noise=true , address=145.79.227.31:4798 , piece_id=QGAJGTG33TYFU5KWXYA74GUYLIO2NWGAHCFAZJUU3GMUH6YARBTA, seg_pos={0 0}, upload=470096]: Done uploading piece. err: <nil> Dec 27 23:29:55.928005119 (536h0m55.799088154s) [noise=true , address=145.79.227.16:4910 , piece_id=HSBDEVHKA7FEMFH2LETPO5PYJIM45RUVPXF4DDNE7WME5BHICWJA, seg_pos={0 0}, upload=470096]: Done uploading piece. err: <nil> Dec 27 23:29:55.936545731 (536h0m55.807628766s) [noise=true , address=145.79.227.36:4518 , piece_id=2V2XOJPMKI4W45JXL2IM2VTH5N5BLMBJK3HLTMXNXNWBAZUOHPJA, seg_pos={0 0}, upload=470096]: Done uploading piece. err: <nil> Dec 27 23:29:55.937277464 (536h0m55.808360499s) [noise=true , address=145.79.227.21:4473 , piece_id=LVJPMVISE4VGQYCF6ZGS5MEISPBR5NRDJYF6MSMVOM2LWWUEMQUA, seg_pos={0 0}, upload=470096]: Done uploading piece. err: <nil> Dec 27 23:29:55.942288466 (536h0m55.813371511s) [noise=true , address=145.79.227.15:4876 , piece_id=GUCTGD6Q5FLPIDQGUA7AFARRRDXPSQGGDKVKRP6ZM6SXZG4LLY2A, seg_pos={0 0}, upload=470096]: Done uploading piece. err: <nil> Dec 27 23:29:55.945849396 (536h0m55.816932432s) [noise=true , address=145.79.227.32:4836 , piece_id=DXS4A2BKZZFTBZTBKVIY2WOOA3YVHGXE5SF3YPC2ITXD6RUCWZ6A, seg_pos={0 0}, upload=470096]: Done uploading piece. err: <nil> Dec 27 23:29:55.951028428 (536h0m55.822111474s) [noise=true , address=145.79.227.37:4567 , piece_id=ZHE7XUOKYHHVPQLW7U3E2BJLTSHGYNXAV65SUC4RIPM4266IWGVA, seg_pos={0 0}, upload=470096]: Done uploading piece. err: <nil> Dec 27 23:29:55.954394818 (536h0m55.825477853s) [noise=true , address=145.79.227.2:4114 , piece_id=JS7UXBFVLPCKSHVZFU2O3NLCMRMPGQUFXLKESLHWS2VCZ4NIWSLA, seg_pos={0 0}, upload=470096]: Done uploading piece. err: <nil> Dec 27 23:29:56.175805030 (536h0m56.046888065s) [noise=true , address=77.90.31.14:4664 , piece_id=GDBOPGPGGZWM4UMG3SSV3PL2IU444Q2CR75RUMYZNFZAQXG3YWCA, seg_pos={0 0}, upload=470096]: Done uploading piece. err: <nil> Dec 27 23:29:56.187177044 (536h0m56.058260079s) [noise=true , address=45.91.116.14:4191 , piece_id=XM3UALWAHCDL6DMO5ZGKE7WG2J4YXINC4DMDFGKPL5BRFGYWI63A, seg_pos={0 0}, upload=470096]: Done uploading piece. err: <nil> Dec 27 23:29:56.216722133 (536h0m56.087805168s) [noise=true , address=45.91.116.7:4010 , piece_id=RDEWAYTVTOMZ6U2PG4GUGTYFSBBMCTA6BW2I5CMHLIL7Z7ED5PLQ, seg_pos={0 0}, upload=470096]: Done uploading piece. err: <nil> Dec 27 23:29:56.342992673 (536h0m56.214075708s) [noise=true , address=45.91.116.13:4156 , piece_id=C5WTVQ772CLEMKG627CU5OTO3A2SXZOQREI6ZOHFHVVPQHWNV7GQ, seg_pos={0 0}, upload=470096]: Done uploading piece. err: <nil> Dec 27 23:29:56.343013814 (536h0m56.214096859s) [seg_pos={0 0}, upload=470096]: Segment reached optimal threshold of 13 pieces. Dec 27 23:29:56.343190420 (536h0m56.214273455s) [noise=true , address=77.90.31.15:4697 , piece_id=ORFKLJSAAG3F2CVTJASOHABK3R74FUTQZDIVOT3FSSSX4Z6Z3KFA, seg_pos={0 0}, upload=470096]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 27 23:29:56.343195840 (536h0m56.214278875s) [noise=true , address=77.90.31.15:4697 , piece_id=ORFKLJSAAG3F2CVTJASOHABK3R74FUTQZDIVOT3FSSSX4Z6Z3KFA, seg_pos={0 0}, upload=470096]: Long tail cancellation due to optimal threshold Dec 27 23:29:56.343367247 (536h0m56.214450282s) [noise=true , address=145.79.227.13:4803 , piece_id=37N7HFRZ5AS3UVBJEZ2BFRZGZUY6DMTMWDQMXZB5FBUK5HEEACHA, seg_pos={0 0}, upload=470096]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7): protocol: write tcp 188.166.225.170:37712->145.79.227.13:4803: use of closed network connection; write tcp 188.166.225.170:37712->145.79.227.13:4803: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37712->145.79.227.13:4803: use of closed network connection Dec 27 23:29:56.343385721 (536h0m56.214468756s) [noise=true , address=145.79.227.13:4803 , piece_id=37N7HFRZ5AS3UVBJEZ2BFRZGZUY6DMTMWDQMXZB5FBUK5HEEACHA, seg_pos={0 0}, upload=470096]: Long tail cancellation due to optimal threshold Dec 27 23:29:56.343393416 (536h0m56.214476461s) [noise=true , address=145.79.227.5:4221 , piece_id=FCU4S2AQPVA6XNWIYR7HN4LB6G5ZNALNO75ZOPSEHOQ5GSJLQGZQ, seg_pos={0 0}, upload=470096]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:42232->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:42232->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42232->145.79.227.5:4221: use of closed network connection Dec 27 23:29:56.343401833 (536h0m56.214484888s) [noise=true , address=145.79.227.5:4221 , piece_id=FCU4S2AQPVA6XNWIYR7HN4LB6G5ZNALNO75ZOPSEHOQ5GSJLQGZQ, seg_pos={0 0}, upload=470096]: Long tail cancellation due to optimal threshold Dec 27 23:29:56.343506091 (536h0m56.214589126s) [seg_pos={0 0}, upload=470096]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:29:56.343592265 (536h0m56.2146753s) [upload=470096]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:29:56.343599098 (536h0m56.214682143s) [upload=470096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:29:56.639685425 (536h0m56.51076846s) [upload=470096]: Done uploading segments... Dec 27 23:29:56.639699893 (536h0m56.510782928s) [upload=470096]: Upload finished. err: <nil> Duration: 30.002156319s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.......X.....X..........................XXX.....XX.XX.XXXX.] duration: 8.154194935s trace: https://jaeger.dev.storj.io/trace/7ed6926dee9a169a FAIL (2025-12-27 23:29:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: OZFONKZMEPFNGOSPYBNX3NATJ4AEVAZOW4TXHPX2UCHJTWJ25HFQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: GAATTMVA7ZPE4VFRGMRGIX6CVEGZG4OXP4RCT3FAXZRNXLC24RLQ) context deadline exceeded ======LOGS====== Dec 27 23:29:05.744144087 (536h0m5.615227132s) [upload=470084]: Starting upload Dec 27 23:29:05.744214371 (536h0m5.615297406s) [upload=470084]: Done starting upload Dec 27 23:29:05.744458686 (536h0m5.615541731s) [upload=470084]: Uploading segments... Dec 27 23:29:05.744502681 (536h0m5.615585726s) [upload=470084]: Got next segment. Inline: false Dec 27 23:29:05.744534802 (536h0m5.615617847s) [upload=470084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:29:05.744540262 (536h0m5.615623307s) [upload=470084]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:29:05.990393036 (536h0m5.861476071s) [seg_pos={0 0}, upload=470084]: Begin upload segment... Dec 27 23:29:05.990442510 (536h0m5.861525555s) [seg_pos={0 0}, upload=470084]: Done begin upload segment. Dec 27 23:29:05.990474652 (536h0m5.861557687s) [noise=true , address=145.79.227.9:4367 , piece_id=6JRXMDV6I5W2EL5KTWYHTV3RIBWUT3QF63TGBPD3ZDGIJVPR5LAQ, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.990484801 (536h0m5.861567836s) [noise=true , address=145.79.227.14:4839 , piece_id=VXYVZZSCOHWYRO4M2UBG4SP26T7VVCUFQDV7RH4VX43MZKPLYTEA, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.990631880 (536h0m5.861714916s) [noise=true , address=45.91.116.12:4226 , piece_id=4QAE5YFKVGC3NFYXORR6YT2T5ZFJXD3GWRXGVXGKT7BB3LGJ4EIQ, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.990920339 (536h0m5.862003385s) [noise=true , address=145.79.227.36:4518 , piece_id=RQQBIUYSACKDCHRWLPFIAI5QM2WN6NBTHZFRKD4NQKBRAXKOBOTQ, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.991193009 (536h0m5.862276044s) [noise=true , address=145.79.227.4:4187 , piece_id=2DJVUC7NG3GC4FN4RQCMPST6M3BPVLBPB2OEQBHYAGNE4PZ66WQQ, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.991542434 (536h0m5.862625479s) [noise=true , address=45.91.116.4:4937 , piece_id=C32JHR6F3ISRA2M5VTSHWAS244UBPU645FJTL33FCQNT54TX2FVQ, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.991906768 (536h0m5.862989814s) [noise=true , address=145.79.227.11:4730 , piece_id=IR3LYYFQ5UE6JXBY56ONJZVC7RJKF4WBINWQSDVGC4IRI7SAFI7Q, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.992346465 (536h0m5.86342951s) [noise=true , address=145.79.227.50:4373 , piece_id=YADZXTLNQMFU4FB6P6HCQF5Q65EILPIZ2XRRZRTHCEQDMQTL4D3Q, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.992771285 (536h0m5.86385433s) [noise=true , address=145.79.227.21:4473 , piece_id=S5XGDUN43B3FXAQ7HMIF52H23TBOYZMPX5LQ5HH6TIY3WAI55HMA, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.993162660 (536h0m5.864245695s) [noise=true , address=45.91.116.7:4012 , piece_id=GAATTMVA7ZPE4VFRGMRGIX6CVEGZG4OXP4RCT3FAXZRNXLC24RLQ, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.998422545 (536h0m5.869505591s) [noise=true , address=45.91.116.14:4190 , piece_id=O2SIFBFXB7YA6I22O435WIORNL47ZTYEVUECZXS5PG5XS3PRH2JA, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:06.000516302 (536h0m5.871599357s) [noise=true , address=145.79.227.48:4303 , piece_id=Q5AUSFUZTR22G4DT3Z7FTASMST6TGGCYB6Y5KQJF6MSLXAHY4NJQ, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:06.003096485 (536h0m5.87417953s) [noise=true , address=145.79.227.41:4086 , piece_id=JKAJNR7EGP7HD57MDHYYVLMRV7JXUZAEKXW2JMNSNBEXKR7I6SWQ, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:06.003747545 (536h0m5.87483059s) [noise=true , address=45.91.116.6:4976 , piece_id=OZFONKZMEPFNGOSPYBNX3NATJ4AEVAZOW4TXHPX2UCHJTWJ25HFQ, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:06.004271954 (536h0m5.875354999s) [noise=true , address=145.79.227.15:4873 , piece_id=JUCOOAY6OMJQURN7S7YVLLBE2GSP34VWZZ3XT75OYZTWPCSP3GNA, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:06.010176546 (536h0m5.881259592s) [noise=true , address=145.79.227.2:4113 , piece_id=NABMY3PYTAQTUFMEIOYHN22THDPGPKTQC4PKR2AMXSQ7LLWCWD6A, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:07.708712129 (536h0m7.579795164s) [upload=470084]: Next returned nil segment Dec 27 23:29:07.708729172 (536h0m7.579812217s) [upload=470084]: Waiting for error group managing segments... Dec 27 23:29:08.001547096 (536h0m7.872630131s) [noise=true , address=145.79.227.36:4518 , piece_id=RQQBIUYSACKDCHRWLPFIAI5QM2WN6NBTHZFRKD4NQKBRAXKOBOTQ, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.005287688 (536h0m7.876370734s) [noise=true , address=145.79.227.15:4873 , piece_id=JUCOOAY6OMJQURN7S7YVLLBE2GSP34VWZZ3XT75OYZTWPCSP3GNA, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.005990247 (536h0m7.877073302s) [noise=true , address=145.79.227.21:4473 , piece_id=S5XGDUN43B3FXAQ7HMIF52H23TBOYZMPX5LQ5HH6TIY3WAI55HMA, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.006455583 (536h0m7.877538628s) [noise=true , address=145.79.227.9:4367 , piece_id=6JRXMDV6I5W2EL5KTWYHTV3RIBWUT3QF63TGBPD3ZDGIJVPR5LAQ, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.007465917 (536h0m7.878548962s) [noise=true , address=145.79.227.14:4839 , piece_id=VXYVZZSCOHWYRO4M2UBG4SP26T7VVCUFQDV7RH4VX43MZKPLYTEA, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.011357537 (536h0m7.882440582s) [noise=true , address=145.79.227.11:4730 , piece_id=IR3LYYFQ5UE6JXBY56ONJZVC7RJKF4WBINWQSDVGC4IRI7SAFI7Q, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.036798039 (536h0m7.907881074s) [noise=true , address=145.79.227.50:4373 , piece_id=YADZXTLNQMFU4FB6P6HCQF5Q65EILPIZ2XRRZRTHCEQDMQTL4D3Q, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.042528732 (536h0m7.913611777s) [noise=true , address=145.79.227.2:4113 , piece_id=NABMY3PYTAQTUFMEIOYHN22THDPGPKTQC4PKR2AMXSQ7LLWCWD6A, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.229772995 (536h0m8.100856031s) [noise=true , address=45.91.116.12:4226 , piece_id=4QAE5YFKVGC3NFYXORR6YT2T5ZFJXD3GWRXGVXGKT7BB3LGJ4EIQ, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.233137823 (536h0m8.104220868s) [noise=true , address=45.91.116.7:4012 , piece_id=GAATTMVA7ZPE4VFRGMRGIX6CVEGZG4OXP4RCT3FAXZRNXLC24RLQ, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.263854412 (536h0m8.134937447s) [noise=true , address=45.91.116.6:4976 , piece_id=OZFONKZMEPFNGOSPYBNX3NATJ4AEVAZOW4TXHPX2UCHJTWJ25HFQ, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.294518302 (536h0m8.165601337s) [noise=true , address=45.91.116.4:4937 , piece_id=C32JHR6F3ISRA2M5VTSHWAS244UBPU645FJTL33FCQNT54TX2FVQ, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.299211038 (536h0m8.170294083s) [noise=true , address=145.79.227.41:4086 , piece_id=JKAJNR7EGP7HD57MDHYYVLMRV7JXUZAEKXW2JMNSNBEXKR7I6SWQ, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.299229693 (536h0m8.170312728s) [seg_pos={0 0}, upload=470084]: Segment reached optimal threshold of 13 pieces. Dec 27 23:29:08.299424544 (536h0m8.170507579s) [noise=true , address=45.91.116.14:4190 , piece_id=O2SIFBFXB7YA6I22O435WIORNL47ZTYEVUECZXS5PG5XS3PRH2JA, seg_pos={0 0}, upload=470084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 23:29:08.299432940 (536h0m8.170515975s) [noise=true , address=45.91.116.14:4190 , piece_id=O2SIFBFXB7YA6I22O435WIORNL47ZTYEVUECZXS5PG5XS3PRH2JA, seg_pos={0 0}, upload=470084]: Long tail cancellation due to optimal threshold Dec 27 23:29:08.299444321 (536h0m8.170527356s) [noise=true , address=145.79.227.48:4303 , piece_id=Q5AUSFUZTR22G4DT3Z7FTASMST6TGGCYB6Y5KQJF6MSLXAHY4NJQ, seg_pos={0 0}, upload=470084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: write tcp 188.166.225.170:55350->145.79.227.48:4303: use of closed network connection; write tcp 188.166.225.170:55350->145.79.227.48:4303: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55350->145.79.227.48:4303: use of closed network connection Dec 27 23:29:08.299452798 (536h0m8.170535833s) [noise=true , address=145.79.227.48:4303 , piece_id=Q5AUSFUZTR22G4DT3Z7FTASMST6TGGCYB6Y5KQJF6MSLXAHY4NJQ, seg_pos={0 0}, upload=470084]: Long tail cancellation due to optimal threshold Dec 27 23:29:08.299622470 (536h0m8.170705505s) [noise=true , address=145.79.227.4:4187 , piece_id=2DJVUC7NG3GC4FN4RQCMPST6M3BPVLBPB2OEQBHYAGNE4PZ66WQQ, seg_pos={0 0}, upload=470084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 27 23:29:08.299628843 (536h0m8.170711878s) [noise=true , address=145.79.227.4:4187 , piece_id=2DJVUC7NG3GC4FN4RQCMPST6M3BPVLBPB2OEQBHYAGNE4PZ66WQQ, seg_pos={0 0}, upload=470084]: Long tail cancellation due to optimal threshold Dec 27 23:29:08.299645845 (536h0m8.17072888s) [seg_pos={0 0}, upload=470084]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:29:08.299711050 (536h0m8.170794085s) [upload=470084]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:29:08.299717832 (536h0m8.170800877s) [upload=470084]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:29:08.586397427 (536h0m8.457480462s) [upload=470084]: Done uploading segments... Dec 27 23:29:08.586448324 (536h0m8.457531369s) [upload=470084]: Upload finished. err: <nil> Duration: 8.154213069s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.......X.....X..........................XXX.....XX.XX.XXXX] duration: 30.005285175s trace: https://jaeger.dev.storj.io/trace/7ed6926dee9a169a FAIL (2025-12-27 23:29:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: OZFONKZMEPFNGOSPYBNX3NATJ4AEVAZOW4TXHPX2UCHJTWJ25HFQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: GAATTMVA7ZPE4VFRGMRGIX6CVEGZG4OXP4RCT3FAXZRNXLC24RLQ) context deadline exceeded ======LOGS====== Dec 27 23:29:05.744144087 (536h0m5.615227132s) [upload=470084]: Starting upload Dec 27 23:29:05.744214371 (536h0m5.615297406s) [upload=470084]: Done starting upload Dec 27 23:29:05.744458686 (536h0m5.615541731s) [upload=470084]: Uploading segments... Dec 27 23:29:05.744502681 (536h0m5.615585726s) [upload=470084]: Got next segment. Inline: false Dec 27 23:29:05.744534802 (536h0m5.615617847s) [upload=470084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:29:05.744540262 (536h0m5.615623307s) [upload=470084]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:29:05.990393036 (536h0m5.861476071s) [seg_pos={0 0}, upload=470084]: Begin upload segment... Dec 27 23:29:05.990442510 (536h0m5.861525555s) [seg_pos={0 0}, upload=470084]: Done begin upload segment. Dec 27 23:29:05.990474652 (536h0m5.861557687s) [noise=true , address=145.79.227.9:4367 , piece_id=6JRXMDV6I5W2EL5KTWYHTV3RIBWUT3QF63TGBPD3ZDGIJVPR5LAQ, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.990484801 (536h0m5.861567836s) [noise=true , address=145.79.227.14:4839 , piece_id=VXYVZZSCOHWYRO4M2UBG4SP26T7VVCUFQDV7RH4VX43MZKPLYTEA, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.990631880 (536h0m5.861714916s) [noise=true , address=45.91.116.12:4226 , piece_id=4QAE5YFKVGC3NFYXORR6YT2T5ZFJXD3GWRXGVXGKT7BB3LGJ4EIQ, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.990920339 (536h0m5.862003385s) [noise=true , address=145.79.227.36:4518 , piece_id=RQQBIUYSACKDCHRWLPFIAI5QM2WN6NBTHZFRKD4NQKBRAXKOBOTQ, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.991193009 (536h0m5.862276044s) [noise=true , address=145.79.227.4:4187 , piece_id=2DJVUC7NG3GC4FN4RQCMPST6M3BPVLBPB2OEQBHYAGNE4PZ66WQQ, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.991542434 (536h0m5.862625479s) [noise=true , address=45.91.116.4:4937 , piece_id=C32JHR6F3ISRA2M5VTSHWAS244UBPU645FJTL33FCQNT54TX2FVQ, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.991906768 (536h0m5.862989814s) [noise=true , address=145.79.227.11:4730 , piece_id=IR3LYYFQ5UE6JXBY56ONJZVC7RJKF4WBINWQSDVGC4IRI7SAFI7Q, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.992346465 (536h0m5.86342951s) [noise=true , address=145.79.227.50:4373 , piece_id=YADZXTLNQMFU4FB6P6HCQF5Q65EILPIZ2XRRZRTHCEQDMQTL4D3Q, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.992771285 (536h0m5.86385433s) [noise=true , address=145.79.227.21:4473 , piece_id=S5XGDUN43B3FXAQ7HMIF52H23TBOYZMPX5LQ5HH6TIY3WAI55HMA, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.993162660 (536h0m5.864245695s) [noise=true , address=45.91.116.7:4012 , piece_id=GAATTMVA7ZPE4VFRGMRGIX6CVEGZG4OXP4RCT3FAXZRNXLC24RLQ, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:05.998422545 (536h0m5.869505591s) [noise=true , address=45.91.116.14:4190 , piece_id=O2SIFBFXB7YA6I22O435WIORNL47ZTYEVUECZXS5PG5XS3PRH2JA, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:06.000516302 (536h0m5.871599357s) [noise=true , address=145.79.227.48:4303 , piece_id=Q5AUSFUZTR22G4DT3Z7FTASMST6TGGCYB6Y5KQJF6MSLXAHY4NJQ, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:06.003096485 (536h0m5.87417953s) [noise=true , address=145.79.227.41:4086 , piece_id=JKAJNR7EGP7HD57MDHYYVLMRV7JXUZAEKXW2JMNSNBEXKR7I6SWQ, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:06.003747545 (536h0m5.87483059s) [noise=true , address=45.91.116.6:4976 , piece_id=OZFONKZMEPFNGOSPYBNX3NATJ4AEVAZOW4TXHPX2UCHJTWJ25HFQ, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:06.004271954 (536h0m5.875354999s) [noise=true , address=145.79.227.15:4873 , piece_id=JUCOOAY6OMJQURN7S7YVLLBE2GSP34VWZZ3XT75OYZTWPCSP3GNA, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:06.010176546 (536h0m5.881259592s) [noise=true , address=145.79.227.2:4113 , piece_id=NABMY3PYTAQTUFMEIOYHN22THDPGPKTQC4PKR2AMXSQ7LLWCWD6A, seg_pos={0 0}, upload=470084]: Uploading piece... Dec 27 23:29:07.708712129 (536h0m7.579795164s) [upload=470084]: Next returned nil segment Dec 27 23:29:07.708729172 (536h0m7.579812217s) [upload=470084]: Waiting for error group managing segments... Dec 27 23:29:08.001547096 (536h0m7.872630131s) [noise=true , address=145.79.227.36:4518 , piece_id=RQQBIUYSACKDCHRWLPFIAI5QM2WN6NBTHZFRKD4NQKBRAXKOBOTQ, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.005287688 (536h0m7.876370734s) [noise=true , address=145.79.227.15:4873 , piece_id=JUCOOAY6OMJQURN7S7YVLLBE2GSP34VWZZ3XT75OYZTWPCSP3GNA, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.005990247 (536h0m7.877073302s) [noise=true , address=145.79.227.21:4473 , piece_id=S5XGDUN43B3FXAQ7HMIF52H23TBOYZMPX5LQ5HH6TIY3WAI55HMA, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.006455583 (536h0m7.877538628s) [noise=true , address=145.79.227.9:4367 , piece_id=6JRXMDV6I5W2EL5KTWYHTV3RIBWUT3QF63TGBPD3ZDGIJVPR5LAQ, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.007465917 (536h0m7.878548962s) [noise=true , address=145.79.227.14:4839 , piece_id=VXYVZZSCOHWYRO4M2UBG4SP26T7VVCUFQDV7RH4VX43MZKPLYTEA, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.011357537 (536h0m7.882440582s) [noise=true , address=145.79.227.11:4730 , piece_id=IR3LYYFQ5UE6JXBY56ONJZVC7RJKF4WBINWQSDVGC4IRI7SAFI7Q, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.036798039 (536h0m7.907881074s) [noise=true , address=145.79.227.50:4373 , piece_id=YADZXTLNQMFU4FB6P6HCQF5Q65EILPIZ2XRRZRTHCEQDMQTL4D3Q, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.042528732 (536h0m7.913611777s) [noise=true , address=145.79.227.2:4113 , piece_id=NABMY3PYTAQTUFMEIOYHN22THDPGPKTQC4PKR2AMXSQ7LLWCWD6A, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.229772995 (536h0m8.100856031s) [noise=true , address=45.91.116.12:4226 , piece_id=4QAE5YFKVGC3NFYXORR6YT2T5ZFJXD3GWRXGVXGKT7BB3LGJ4EIQ, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.233137823 (536h0m8.104220868s) [noise=true , address=45.91.116.7:4012 , piece_id=GAATTMVA7ZPE4VFRGMRGIX6CVEGZG4OXP4RCT3FAXZRNXLC24RLQ, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.263854412 (536h0m8.134937447s) [noise=true , address=45.91.116.6:4976 , piece_id=OZFONKZMEPFNGOSPYBNX3NATJ4AEVAZOW4TXHPX2UCHJTWJ25HFQ, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.294518302 (536h0m8.165601337s) [noise=true , address=45.91.116.4:4937 , piece_id=C32JHR6F3ISRA2M5VTSHWAS244UBPU645FJTL33FCQNT54TX2FVQ, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.299211038 (536h0m8.170294083s) [noise=true , address=145.79.227.41:4086 , piece_id=JKAJNR7EGP7HD57MDHYYVLMRV7JXUZAEKXW2JMNSNBEXKR7I6SWQ, seg_pos={0 0}, upload=470084]: Done uploading piece. err: <nil> Dec 27 23:29:08.299229693 (536h0m8.170312728s) [seg_pos={0 0}, upload=470084]: Segment reached optimal threshold of 13 pieces. Dec 27 23:29:08.299424544 (536h0m8.170507579s) [noise=true , address=45.91.116.14:4190 , piece_id=O2SIFBFXB7YA6I22O435WIORNL47ZTYEVUECZXS5PG5XS3PRH2JA, seg_pos={0 0}, upload=470084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 27 23:29:08.299432940 (536h0m8.170515975s) [noise=true , address=45.91.116.14:4190 , piece_id=O2SIFBFXB7YA6I22O435WIORNL47ZTYEVUECZXS5PG5XS3PRH2JA, seg_pos={0 0}, upload=470084]: Long tail cancellation due to optimal threshold Dec 27 23:29:08.299444321 (536h0m8.170527356s) [noise=true , address=145.79.227.48:4303 , piece_id=Q5AUSFUZTR22G4DT3Z7FTASMST6TGGCYB6Y5KQJF6MSLXAHY4NJQ, seg_pos={0 0}, upload=470084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gJKrCBLJWN1SBKibfeqBtcqqiv4veKTGmtnycjZyb3VN9kwoT): protocol: write tcp 188.166.225.170:55350->145.79.227.48:4303: use of closed network connection; write tcp 188.166.225.170:55350->145.79.227.48:4303: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55350->145.79.227.48:4303: use of closed network connection Dec 27 23:29:08.299452798 (536h0m8.170535833s) [noise=true , address=145.79.227.48:4303 , piece_id=Q5AUSFUZTR22G4DT3Z7FTASMST6TGGCYB6Y5KQJF6MSLXAHY4NJQ, seg_pos={0 0}, upload=470084]: Long tail cancellation due to optimal threshold Dec 27 23:29:08.299622470 (536h0m8.170705505s) [noise=true , address=145.79.227.4:4187 , piece_id=2DJVUC7NG3GC4FN4RQCMPST6M3BPVLBPB2OEQBHYAGNE4PZ66WQQ, seg_pos={0 0}, upload=470084]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: expected piece hash; context canceled; EOF Dec 27 23:29:08.299628843 (536h0m8.170711878s) [noise=true , address=145.79.227.4:4187 , piece_id=2DJVUC7NG3GC4FN4RQCMPST6M3BPVLBPB2OEQBHYAGNE4PZ66WQQ, seg_pos={0 0}, upload=470084]: Long tail cancellation due to optimal threshold Dec 27 23:29:08.299645845 (536h0m8.17072888s) [seg_pos={0 0}, upload=470084]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:29:08.299711050 (536h0m8.170794085s) [upload=470084]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:29:08.299717832 (536h0m8.170800877s) [upload=470084]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:29:08.586397427 (536h0m8.457480462s) [upload=470084]: Done uploading segments... Dec 27 23:29:08.586448324 (536h0m8.457531369s) [upload=470084]: Upload finished. err: <nil> Duration: 30.005307797s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.430762757s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:29:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430802071s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.......X.....X..........................XXX.....XX.XX.XXX] duration: 30.002293903s trace: https://jaeger.dev.storj.io/trace/4415c0cb0d3c375e FAIL (2025-12-27 23:28:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: MFFF3NFUCSAM6ZCD3J63JRHZJZ3UFTINE7J5JKS7HLREYSIY2REA) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: GN5JTTZSA34KMEDLTJTU5VLXTYBZS27VHQ5OG6I65XBFI67SN7BQ) context deadline exceeded ======LOGS====== Dec 27 23:28:28.772963920 (535h59m28.644046955s) [upload=470076]: Starting upload Dec 27 23:28:28.773014256 (535h59m28.644097291s) [upload=470076]: Done starting upload Dec 27 23:28:28.773165413 (535h59m28.644248448s) [upload=470076]: Uploading segments... Dec 27 23:28:28.773273078 (535h59m28.644356113s) [upload=470076]: Got next segment. Inline: false Dec 27 23:28:28.773314768 (535h59m28.644397803s) [upload=470076]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:28:28.773320850 (535h59m28.644403885s) [upload=470076]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:28:29.023832054 (535h59m28.894915109s) [seg_pos={0 0}, upload=470076]: Begin upload segment... Dec 27 23:28:29.023900394 (535h59m28.894983469s) [seg_pos={0 0}, upload=470076]: Done begin upload segment. Dec 27 23:28:29.023967081 (535h59m28.895050136s) [noise=true , address=145.79.227.16:4912 , piece_id=2ZAHCOPEWRARSV2QNQ2NI6D6L4RMAWJJUBGVWP35QIEETQF6ICYA, seg_pos={0 0}, upload=470076]: Uploading piece... Dec 27 23:28:29.023988983 (535h59m28.895072018s) [noise=true , address=145.79.227.3:4151 , piece_id=H7PZZ4KFEYHSKQJOPIOIDL2XXMGENH7Y63V3ENI33QMFGJE6IM2Q, seg_pos={0 0}, upload=470076]: Uploading piece... Dec 27 23:28:29.024713602 (535h59m28.895796637s) [noise=true , address=45.91.116.5:4297 , piece_id=JZVVPWUOIMNMSEVJHDODYUS267TK54ZK7W2GKUL4VZC3DPML635A, seg_pos={0 0}, upload=470076]: Uploading piece... Dec 27 23:28:29.024959371 (535h59m28.896042426s) [noise=true , address=45.91.116.9:4262 , piece_id=4YVLVTI7YFTZXJBOHQVP2GIFMQSG334FLAYSEVMOZWEK7QMDOJBA, seg_pos={0 0}, upload=470076]: Uploading piece... Dec 27 23:28:29.025522002 (535h59m28.896605047s) [noise=true , address=45.91.116.7:4010 , piece_id=AOOOFJXRLPGKCDM7TJWJS3MGOPENW34OUESU2H7PJDTKIDKHNKMQ, seg_pos={0 0}, upload=470076]: Uploading piece... Dec 27 23:28:29.026034598 (535h59m28.897117633s) [noise=true , address=45.91.116.13:4154 , piece_id=QCRPFLZCZOJNK3UVCRDHHCBGWSWNAVIA2PZU3SRFN7VK4GLUUIFA, seg_pos={0 0}, upload=470076]: Uploading piece... Dec 27 23:28:29.026331163 (535h59m28.897414198s) [noise=true , address=45.91.116.12:4227 , piece_id=YTTTC447OMLJYDYGMFRIGSSMXEKW7XMNDM7BSQDGF5E2ZOXUMYBQ, seg_pos={0 0}, upload=470076]: Uploading piece... Dec 27 23:28:29.026753587 (535h59m28.897836622s) [noise=true , address=77.90.31.18:4808 , piece_id=MFFF3NFUCSAM6ZCD3J63JRHZJZ3UFTINE7J5JKS7HLREYSIY2REA, seg_pos={0 0}, upload=470076]: Uploading piece... Dec 27 23:28:29.026865159 (535h59m28.897948194s) [noise=true , address=145.79.227.7:4293 , piece_id=GB752II3TEPCYTKRW6PMWLCADNPBFGE4NWT7VEQ4UGE4AWQUXMRQ, seg_pos={0 0}, upload=470076]: Uploading piece... Dec 27 23:28:29.027298455 (535h59m28.89838154s) [noise=true , address=45.91.116.11:4118 , piece_id=E5IDWA2AD25EDJCJ5MF2OHAZZR74EVE6BLJJTA33OHVERYM66RIA, seg_pos={0 0}, upload=470076]: Uploading piece... Dec 27 23:28:29.027381012 (535h59m28.898464057s) [noise=true , address=77.90.31.19:4841 , piece_id=LDGEXCAJZOTLKBO3PSWI5BUKEUQIJ3W75KSWODQ3PG2EMO7L4THQ, seg_pos={0 0}, upload=470076]: Uploading piece... Dec 27 23:28:29.027892586 (535h59m28.898975621s) [noise=true , address=45.91.116.10:4082 , piece_id=OI2XGPDAW3YPG6VRHC6ZF6EMP22EBVP64DVLWF45GLRQCC6KMCSA, seg_pos={0 0}, upload=470076]: Uploading piece... Dec 27 23:28:29.028054955 (535h59m28.899138s) [noise=true , address=145.79.227.25:4618 , piece_id=26CDSNL2DPQ3XLI5I4UPCDO4D3SPJ2J6KPKONPPTOHNGHWRMN3CA, seg_pos={0 0}, upload=470076]: Uploading piece... Dec 27 23:28:29.028398098 (535h59m28.899481153s) [noise=true , address=145.79.227.47:4265 , piece_id=2QYHYDTDOTS5WKT2CN5XEOCT36ITRYCG4A2IZNEA24M7L3SZQNBA, seg_pos={0 0}, upload=470076]: Uploading piece... Dec 27 23:28:29.028557202 (535h59m28.899640257s) [noise=true , address=45.91.116.6:4974 , piece_id=GN5JTTZSA34KMEDLTJTU5VLXTYBZS27VHQ5OG6I65XBFI67SN7BQ, seg_pos={0 0}, upload=470076]: Uploading piece... Dec 27 23:28:29.029059137 (535h59m28.900142182s) [noise=true , address=77.90.31.14:4662 , piece_id=YWUTYP5YPKGAWMOVICFZOU3DAJTPQ3SZCXKZKHZBMYDVJYUAF3RA, seg_pos={0 0}, upload=470076]: Uploading piece... Dec 27 23:28:30.749551599 (535h59m30.620634634s) [upload=470076]: Next returned nil segment Dec 27 23:28:30.749565426 (535h59m30.620648461s) [upload=470076]: Waiting for error group managing segments... Dec 27 23:28:31.028474324 (535h59m30.899557359s) [noise=true , address=145.79.227.7:4293 , piece_id=GB752II3TEPCYTKRW6PMWLCADNPBFGE4NWT7VEQ4UGE4AWQUXMRQ, seg_pos={0 0}, upload=470076]: Done uploading piece. err: <nil> Dec 27 23:28:31.035085181 (535h59m30.906168216s) [noise=true , address=145.79.227.3:4151 , piece_id=H7PZZ4KFEYHSKQJOPIOIDL2XXMGENH7Y63V3ENI33QMFGJE6IM2Q, seg_pos={0 0}, upload=470076]: Done uploading piece. err: <nil> Dec 27 23:28:31.040417095 (535h59m30.91150014s) [noise=true , address=145.79.227.16:4912 , piece_id=2ZAHCOPEWRARSV2QNQ2NI6D6L4RMAWJJUBGVWP35QIEETQF6ICYA, seg_pos={0 0}, upload=470076]: Done uploading piece. err: <nil> Dec 27 23:28:31.050589004 (535h59m30.921672039s) [noise=true , address=145.79.227.47:4265 , piece_id=2QYHYDTDOTS5WKT2CN5XEOCT36ITRYCG4A2IZNEA24M7L3SZQNBA, seg_pos={0 0}, upload=470076]: Done uploading piece. err: <nil> Dec 27 23:28:31.057779315 (535h59m30.92886235s) [noise=true , address=145.79.227.25:4618 , piece_id=26CDSNL2DPQ3XLI5I4UPCDO4D3SPJ2J6KPKONPPTOHNGHWRMN3CA, seg_pos={0 0}, upload=470076]: Done uploading piece. err: <nil> Dec 27 23:28:31.207104525 (535h59m31.07818756s) [noise=true , address=77.90.31.19:4841 , piece_id=LDGEXCAJZOTLKBO3PSWI5BUKEUQIJ3W75KSWODQ3PG2EMO7L4THQ, seg_pos={0 0}, upload=470076]: Done uploading piece. err: <nil> Dec 27 23:28:31.260770053 (535h59m31.131853088s) [noise=true , address=45.91.116.13:4154 , piece_id=QCRPFLZCZOJNK3UVCRDHHCBGWSWNAVIA2PZU3SRFN7VK4GLUUIFA, seg_pos={0 0}, upload=470076]: Done uploading piece. err: <nil> Dec 27 23:28:31.292102666 (535h59m31.163185711s) [noise=true , address=45.91.116.6:4974 , piece_id=GN5JTTZSA34KMEDLTJTU5VLXTYBZS27VHQ5OG6I65XBFI67SN7BQ, seg_pos={0 0}, upload=470076]: Done uploading piece. err: <nil> Dec 27 23:28:31.307030140 (535h59m31.178113186s) [noise=true , address=45.91.116.5:4297 , piece_id=JZVVPWUOIMNMSEVJHDODYUS267TK54ZK7W2GKUL4VZC3DPML635A, seg_pos={0 0}, upload=470076]: Done uploading piece. err: <nil> Dec 27 23:28:31.317229452 (535h59m31.188312487s) [noise=true , address=77.90.31.18:4808 , piece_id=MFFF3NFUCSAM6ZCD3J63JRHZJZ3UFTINE7J5JKS7HLREYSIY2REA, seg_pos={0 0}, upload=470076]: Done uploading piece. err: <nil> Dec 27 23:28:31.324631316 (535h59m31.195714361s) [noise=true , address=45.91.116.10:4082 , piece_id=OI2XGPDAW3YPG6VRHC6ZF6EMP22EBVP64DVLWF45GLRQCC6KMCSA, seg_pos={0 0}, upload=470076]: Done uploading piece. err: <nil> Dec 27 23:28:31.343934171 (535h59m31.215017206s) [noise=true , address=45.91.116.11:4118 , piece_id=E5IDWA2AD25EDJCJ5MF2OHAZZR74EVE6BLJJTA33OHVERYM66RIA, seg_pos={0 0}, upload=470076]: Done uploading piece. err: <nil> Dec 27 23:28:31.346821329 (535h59m31.217904364s) [noise=true , address=45.91.116.12:4227 , piece_id=YTTTC447OMLJYDYGMFRIGSSMXEKW7XMNDM7BSQDGF5E2ZOXUMYBQ, seg_pos={0 0}, upload=470076]: Done uploading piece. err: <nil> Dec 27 23:28:31.346843101 (535h59m31.217926136s) [seg_pos={0 0}, upload=470076]: Segment reached optimal threshold of 13 pieces. Dec 27 23:28:31.347007033 (535h59m31.218090078s) [noise=true , address=45.91.116.9:4262 , piece_id=4YVLVTI7YFTZXJBOHQVP2GIFMQSG334FLAYSEVMOZWEK7QMDOJBA, seg_pos={0 0}, upload=470076]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:53646->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:53646->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53646->45.91.116.9:4262: use of closed network connection Dec 27 23:28:31.347013154 (535h59m31.2180962s) [noise=true , address=45.91.116.9:4262 , piece_id=4YVLVTI7YFTZXJBOHQVP2GIFMQSG334FLAYSEVMOZWEK7QMDOJBA, seg_pos={0 0}, upload=470076]: Long tail cancellation due to optimal threshold Dec 27 23:28:31.347125428 (535h59m31.218208463s) [noise=true , address=77.90.31.14:4662 , piece_id=YWUTYP5YPKGAWMOVICFZOU3DAJTPQ3SZCXKZKHZBMYDVJYUAF3RA, seg_pos={0 0}, upload=470076]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Dec 27 23:28:31.347131159 (535h59m31.218214204s) [noise=true , address=77.90.31.14:4662 , piece_id=YWUTYP5YPKGAWMOVICFZOU3DAJTPQ3SZCXKZKHZBMYDVJYUAF3RA, seg_pos={0 0}, upload=470076]: Long tail cancellation due to optimal threshold Dec 27 23:28:31.347156367 (535h59m31.218239412s) [noise=true , address=45.91.116.7:4010 , piece_id=AOOOFJXRLPGKCDM7TJWJS3MGOPENW34OUESU2H7PJDTKIDKHNKMQ, seg_pos={0 0}, upload=470076]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 23:28:31.347172497 (535h59m31.218255532s) [noise=true , address=45.91.116.7:4010 , piece_id=AOOOFJXRLPGKCDM7TJWJS3MGOPENW34OUESU2H7PJDTKIDKHNKMQ, seg_pos={0 0}, upload=470076]: Long tail cancellation due to optimal threshold Dec 27 23:28:31.347210540 (535h59m31.218293575s) [seg_pos={0 0}, upload=470076]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:28:31.347335077 (535h59m31.218418112s) [upload=470076]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:28:31.347340638 (535h59m31.218423673s) [upload=470076]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:28:31.640488209 (535h59m31.511571284s) [upload=470076]: Done uploading segments... Dec 27 23:28:31.640513608 (535h59m31.511596653s) [upload=470076]: Upload finished. err: <nil> Duration: 30.002312447s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.548507613s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:28:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.548534855s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.520400705s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:28:03 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520428968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X.......X.....X..........................XXX.....XX.XX.XX] duration: 30.002495199s trace: https://jaeger.dev.storj.io/trace/7333926fb2fe50b FAIL (2025-12-27 23:27:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s... Dec 27 23:27:20.041833952 (535h58m19.912916987s) [upload=470060]: Got next segment. Inline: false Dec 27 23:27:20.041891100 (535h58m19.912974145s) [upload=470060]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:27:20.041896680 (535h58m19.912979715s) [upload=470060]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:27:20.282010817 (535h58m20.153093852s) [seg_pos={0 0}, upload=470060]: Begin upload segment... Dec 27 23:27:20.282053759 (535h58m20.153136794s) [seg_pos={0 0}, upload=470060]: Done begin upload segment. Dec 27 23:27:20.282090569 (535h58m20.153173604s) [noise=true , address=145.79.227.17:4972 , piece_id=KFGTEVZEFH4ZABDGAMS2V6WS5JUXJTQXIZVP34ABHI7JDUMVRDYQ, seg_pos={0 0}, upload=470060]: Uploading piece... Dec 27 23:27:20.282100098 (535h58m20.153183133s) [noise=true , address=45.91.116.12:4226 , piece_id=QXDBM3DSBQNQ6VJ6QEYGODKYPOEGU2HFVSU34Q25V5LIRU27TZMQ, seg_pos={0 0}, upload=470060]: Uploading piece... Dec 27 23:27:20.282906363 (535h58m20.153989398s) [noise=true , address=145.79.227.11:4731 , piece_id=M2U3UN2JNZNQ7CQE573N7QPW24PG7WQKIAEPKLSV6VQI65QFXXGQ, seg_pos={0 0}, upload=470060]: Uploading piece... Dec 27 23:27:20.283315653 (535h58m20.154398688s) [noise=true , address=145.79.227.2:4114 , piece_id=TZYAAOGJBILHZSULWJ3B2A6IEQZMZQLUUQVPXE56KPACVCEGQXEA, seg_pos={0 0}, upload=470060]: Uploading piece... Dec 27 23:27:20.283478432 (535h58m20.154561467s) [noise=true , address=145.79.227.15:4876 , piece_id=KX5QXYPTFTEOYUAQ5BBFZVZJVUUHIZPC64W57DNPYR4BBGHVP6NA, seg_pos={0 0}, upload=470060]: Uploading piece... Dec 27 23:27:20.283698532 (535h58m20.154781567s) [noise=true , address=145.79.227.47:4267 , piece_id=VKHYEVC35QC66YLXA7CWJV5LZ6LFPYFZV72POGCQR2RSBQVX62LQ, seg_pos={0 0}, upload=470060]: Uploading piece... Dec 27 23:27:20.283814703 (535h58m20.154897738s) [noise=true , address=145.79.227.36:4517 , piece_id=KGOZG22CIMCE6NFRNO5A4W3VZFTMSMYSA2PIYKSCABMP7V4KETBQ, seg_pos={0 0}, upload=470060]: Uploading piece... Dec 27 23:27:20.284066973 (535h58m20.155150018s) [noise=true , address=145.79.227.48:4303 , piece_id=JQ52Z7XXTPDZI2FI5UXLRXVN4VOSJHHJQD2VZLQOGHCZJMOEEFMA, seg_pos={0 0}, upload=470060]: Uploading piece... Dec 27 23:27:20.284102350 (535h58m20.155185385s) [noise=true , address=145.79.227.9:4365 , piece_id=3UQY45PELQYLK2DZBOQDVH464DHASEY3YTAO7HPTVU6QVHPJM4FA, seg_pos={0 0}, upload=470060]: Uploading piece... Dec 27 23:27:20.284420345 (535h58m20.15550338s) [noise=true , address=77.90.31.14:4662 , piece_id=LPDVOM2XPGVYLQD7FJSU2FIU2GUET2JFRF2HXZGVAXYZJ7QIP77A, seg_pos={0 0}, upload=470060]: Uploading piece... Dec 27 23:27:20.284504717 (535h58m20.155587752s) [noise=true , address=45.91.116.13:4154 , piece_id=RF7JUFDILSVUKAIS77ZUHVG5HCSPAP34N6JBGWZIYFAM55PQ3P2A, seg_pos={0 0}, upload=470060]: Uploading piece... Dec 27 23:27:20.284788918 (535h58m20.155871953s) [noise=true , address=145.79.227.31:4798 , piece_id=JH6EDHKHHZNBLQPKKYQYSRN645TBIGS6QP4UASU352FK5MABOS3A, seg_pos={0 0}, upload=470060]: Uploading piece... Dec 27 23:27:20.284796762 (535h58m20.155879797s) [noise=true , address=145.79.227.52:4447 , piece_id=PBMS7VMMRHIIQ3KOII3X7WA6DFDFNKN4DZPR6TFGYFDC7B5EKZJQ, seg_pos={0 0}, upload=470060]: Uploading piece... Dec 27 23:27:20.285066797 (535h58m20.156149832s) [noise=true , address=145.79.227.16:4911 , piece_id=T6Q7XWPG7MEMCCOBONUQXPW3KGATFGI43M3RSW5E6ANXRH5QZNPA, seg_pos={0 0}, upload=470060]: Uploading piece... Dec 27 23:27:20.285382017 (535h58m20.156465052s) [noise=true , address=145.79.227.5:4223 , piece_id=LIS5Y5KAKBP6QZ25PTUPGTXI44R2XM6ZUJYGXVLAWRY4AKKJT4LQ, seg_pos={0 0}, upload=470060]: Uploading piece... Dec 27 23:27:20.286586331 (535h58m20.157669366s) [noise=true , address=45.91.116.14:4190 , piece_id=H67ZIUMQ35GD7RXRJDXXLSU5M4OHHJTM6XVE6D75JZF6AF7PS43Q, seg_pos={0 0}, upload=470060]: Uploading piece... Dec 27 23:27:21.989145314 (535h58m21.860228349s) [upload=470060]: Next returned nil segment Dec 27 23:27:21.989157667 (535h58m21.860240712s) [upload=470060]: Waiting for error group managing segments... Dec 27 23:27:22.272604075 (535h58m22.14368712s) [noise=true , address=145.79.227.9:4365 , piece_id=3UQY45PELQYLK2DZBOQDVH464DHASEY3YTAO7HPTVU6QVHPJM4FA, seg_pos={0 0}, upload=470060]: Done uploading piece. err: <nil> Dec 27 23:27:22.285317794 (535h58m22.156400849s) [noise=true , address=145.79.227.52:4447 , piece_id=PBMS7VMMRHIIQ3KOII3X7WA6DFDFNKN4DZPR6TFGYFDC7B5EKZJQ, seg_pos={0 0}, upload=470060]: Done uploading piece. err: <nil> Dec 27 23:27:22.287527693 (535h58m22.158610728s) [noise=true , address=145.79.227.36:4517 , piece_id=KGOZG22CIMCE6NFRNO5A4W3VZFTMSMYSA2PIYKSCABMP7V4KETBQ, seg_pos={0 0}, upload=470060]: Done uploading piece. err: <nil> Dec 27 23:27:22.291119282 (535h58m22.162202337s) [noise=true , address=145.79.227.2:4114 , piece_id=TZYAAOGJBILHZSULWJ3B2A6IEQZMZQLUUQVPXE56KPACVCEGQXEA, seg_pos={0 0}, upload=470060]: Done uploading piece. err: <nil> Dec 27 23:27:22.299745037 (535h58m22.170828082s) [noise=true , address=145.79.227.48:4303 , piece_id=JQ52Z7XXTPDZI2FI5UXLRXVN4VOSJHHJQD2VZLQOGHCZJMOEEFMA, seg_pos={0 0}, upload=470060]: Done uploading piece. err: <nil> Dec 27 23:27:22.303808224 (535h58m22.174891259s) [noise=true , address=145.79.227.31:4798 , piece_id=JH6EDHKHHZNBLQPKKYQYSRN645TBIGS6QP4UASU352FK5MABOS3A, seg_pos={0 0}, upload=470060]: Done uploading piece. err: <nil> Dec 27 23:27:22.304185051 (535h58m22.175268086s) [noise=true , address=145.79.227.11:4731 , piece_id=M2U3UN2JNZNQ7CQE573N7QPW24PG7WQKIAEPKLSV6VQI65QFXXGQ, seg_pos={0 0}, upload=470060]: Done uploading piece. err: <nil> Dec 27 23:27:22.309529358 (535h58m22.180612393s) [noise=true , address=145.79.227.16:4911 , piece_id=T6Q7XWPG7MEMCCOBONUQXPW3KGATFGI43M3RSW5E6ANXRH5QZNPA, seg_pos={0 0}, upload=470060]: Done uploading piece. err: <nil> Dec 27 23:27:22.309900385 (535h58m22.18098343s) [noise=true , address=145.79.227.5:4223 , piece_id=LIS5Y5KAKBP6QZ25PTUPGTXI44R2XM6ZUJYGXVLAWRY4AKKJT4LQ, seg_pos={0 0}, upload=470060]: Done uploading piece. err: <nil> Dec 27 23:27:22.310507119 (535h58m22.181590155s) [noise=true , address=145.79.227.47:4267 , piece_id=VKHYEVC35QC66YLXA7CWJV5LZ6LFPYFZV72POGCQR2RSBQVX62LQ, seg_pos={0 0}, upload=470060]: Done uploading piece. err: <nil> Dec 27 23:27:22.319987942 (535h58m22.191070977s) [noise=true , address=145.79.227.17:4972 , piece_id=KFGTEVZEFH4ZABDGAMS2V6WS5JUXJTQXIZVP34ABHI7JDUMVRDYQ, seg_pos={0 0}, upload=470060]: Done uploading piece. err: <nil> Dec 27 23:27:22.485934352 (535h58m22.357017387s) [noise=true , address=45.91.116.14:4190 , piece_id=H67ZIUMQ35GD7RXRJDXXLSU5M4OHHJTM6XVE6D75JZF6AF7PS43Q, seg_pos={0 0}, upload=470060]: Done uploading piece. err: <nil> Dec 27 23:27:22.530413618 (535h58m22.401496653s) [noise=true , address=77.90.31.14:4662 , piece_id=LPDVOM2XPGVYLQD7FJSU2FIU2GUET2JFRF2HXZGVAXYZJ7QIP77A, seg_pos={0 0}, upload=470060]: Done uploading piece. err: <nil> Dec 27 23:27:22.530432454 (535h58m22.401515489s) [seg_pos={0 0}, upload=470060]: Segment reached optimal threshold of 13 pieces. Dec 27 23:27:22.530623458 (535h58m22.401706493s) [noise=true , address=45.91.116.12:4226 , piece_id=QXDBM3DSBQNQ6VJ6QEYGODKYPOEGU2HFVSU34Q25V5LIRU27TZMQ, seg_pos={0 0}, upload=470060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 27 23:27:22.530630351 (535h58m22.401713386s) [noise=true , address=45.91.116.12:4226 , piece_id=QXDBM3DSBQNQ6VJ6QEYGODKYPOEGU2HFVSU34Q25V5LIRU27TZMQ, seg_pos={0 0}, upload=470060]: Long tail cancellation due to optimal threshold Dec 27 23:27:22.530842324 (535h58m22.401925359s) [noise=true , address=45.91.116.13:4154 , piece_id=RF7JUFDILSVUKAIS77ZUHVG5HCSPAP34N6JBGWZIYFAM55PQ3P2A, seg_pos={0 0}, upload=470060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: write tcp 188.166.225.170:44752->45.91.116.13:4154: use of closed network connection; write tcp 188.166.225.170:44752->45.91.116.13:4154: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44752->45.91.116.13:4154: use of closed network connection Dec 27 23:27:22.530858646 (535h58m22.401941681s) [noise=true , address=45.91.116.13:4154 , piece_id=RF7JUFDILSVUKAIS77ZUHVG5HCSPAP34N6JBGWZIYFAM55PQ3P2A, seg_pos={0 0}, upload=470060]: Long tail cancellation due to optimal threshold Dec 27 23:27:22.530929771 (535h58m22.402012806s) [noise=true , address=145.79.227.15:4876 , piece_id=KX5QXYPTFTEOYUAQ5BBFZVZJVUUHIZPC64W57DNPYR4BBGHVP6NA, seg_pos={0 0}, upload=470060]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:36638->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:36638->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36638->145.79.227.15:4876: use of closed network connection Dec 27 23:27:22.530937045 (535h58m22.40202009s) [noise=true , address=145.79.227.15:4876 , piece_id=KX5QXYPTFTEOYUAQ5BBFZVZJVUUHIZPC64W57DNPYR4BBGHVP6NA, seg_pos={0 0}, upload=470060]: Long tail cancellation due to optimal threshold Dec 27 23:27:22.530958375 (535h58m22.40204141s) [seg_pos={0 0}, upload=470060]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:27:22.531081219 (535h58m22.402164264s) [upload=470060]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:27:22.531086900 (535h58m22.402169935s) [upload=470060]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:27:22.821296157 (535h58m22.692379262s) [upload=470060]: Done uploading segments... Dec 27 23:27:22.821318770 (535h58m22.692401805s) [upload=470060]: Upload finished. err: <nil> Duration: 30.002515127s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.500055123s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:27:30 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.500083938s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X.......X.....X..........................XXX.....XX.XX.X] duration: 30.003282984s trace: https://jaeger.dev.storj.io/trace/503a3c78cc8bd4e7 FAIL (2025-12-27 23:26:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: YU6FIY5TJBTVS5673X66XUKXX3KGWELIWOG4IALNZK2Z2XG6MPYA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: JW5F6KSZ7F6HTRDYJPRNJE2S7EJSXI66CKTH7JULYWHPNB3EF5MA) context deadline exceeded ======LOGS====== Dec 27 23:26:09.757271274 (535h57m9.628354309s) [upload=470044]: Starting upload Dec 27 23:26:09.757334986 (535h57m9.628418021s) [upload=470044]: Done starting upload Dec 27 23:26:09.757427912 (535h57m9.628510947s) [upload=470044]: Uploading segments... Dec 27 23:26:09.757531359 (535h57m9.628614404s) [upload=470044]: Got next segment. Inline: false Dec 27 23:26:09.757566185 (535h57m9.62864923s) [upload=470044]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:26:09.757571705 (535h57m9.628654741s) [upload=470044]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:26:10.002773501 (535h57m9.873856556s) [seg_pos={0 0}, upload=470044]: Begin upload segment... Dec 27 23:26:10.002846390 (535h57m9.873929435s) [seg_pos={0 0}, upload=470044]: Done begin upload segment. Dec 27 23:26:10.002892698 (535h57m9.873975743s) [noise=true , address=145.79.227.36:4518 , piece_id=5Y24USRES4CGVFJLIKQMEN6B47PTLVGH7KTGFOBAHH2P4INIQIBA, seg_pos={0 0}, upload=470044]: Uploading piece... Dec 27 23:26:10.002939977 (535h57m9.874023022s) [noise=true , address=45.91.116.8:4047 , piece_id=KICSFLOMS4GPZGGADAGVRVUVYFRYACRHAYFKHPZKMLQVCUU3ZFSA, seg_pos={0 0}, upload=470044]: Uploading piece... Dec 27 23:26:10.003565398 (535h57m9.874648443s) [noise=true , address=45.91.116.14:4189 , piece_id=XZVQTGPHBUH243NCG67L33UFBN65DWKNBBL5QRU3RKHPR7ORNWVQ, seg_pos={0 0}, upload=470044]: Uploading piece... Dec 27 23:26:10.004011658 (535h57m9.875094693s) [noise=true , address=145.79.227.37:4565 , piece_id=SMAAEQIVIAKKWWZVFJF3BXJVKRNBAEMJOKKXRJVEETKPV2RM3WHA, seg_pos={0 0}, upload=470044]: Uploading piece... Dec 27 23:26:10.004373588 (535h57m9.875456633s) [noise=true , address=45.91.116.11:4118 , piece_id=ZCYN7MKSCJ4TH3OALKIASCCW7AIGJO76NNAFT3R75EPWROMIDTQQ, seg_pos={0 0}, upload=470044]: Uploading piece... Dec 27 23:26:10.004742760 (535h57m9.875825805s) [noise=true , address=145.79.227.11:4731 , piece_id=JI2OU4IDEBRUYAKDKWFVCMB7F4ZVE7XHZDA6OGTLSTGGGNRLRYUA, seg_pos={0 0}, upload=470044]: Uploading piece... Dec 27 23:26:10.005081606 (535h57m9.876164651s) [noise=true , address=145.79.227.39:4015 , piece_id=SCNEGZ2PEZKRZCZG37G4GZEXJPQ5BNIVLRKJBLLOMDJUTGSUVNUQ, seg_pos={0 0}, upload=470044]: Uploading piece... Dec 27 23:26:10.005438115 (535h57m9.87652116s) [noise=true , address=145.79.227.2:4114 , piece_id=VP5LKMP3SI6ACXAJRE67AZVX7QSZIYSEHMOB4F4MOYMITNUTMUOQ, seg_pos={0 0}, upload=470044]: Uploading piece... Dec 27 23:26:10.005895075 (535h57m9.87697812s) [noise=true , address=77.90.31.10:4520 , piece_id=YU6FIY5TJBTVS5673X66XUKXX3KGWELIWOG4IALNZK2Z2XG6MPYA, seg_pos={0 0}, upload=470044]: Uploading piece... Dec 27 23:26:10.007915161 (535h57m9.878998196s) [noise=true , address=45.91.116.5:4297 , piece_id=WAUKVMZTOMP6SCUH4TKPJMTMNAMCCYXU54VUWHOA3SKRGJ2OOPSQ, seg_pos={0 0}, upload=470044]: Uploading piece... Dec 27 23:26:10.009407272 (535h57m9.880490308s) [noise=true , address=45.91.116.10:4083 , piece_id=JW5F6KSZ7F6HTRDYJPRNJE2S7EJSXI66CKTH7JULYWHPNB3EF5MA, seg_pos={0 0}, upload=470044]: Uploading piece... Dec 27 23:26:10.011106839 (535h57m9.882189874s) [noise=true , address=145.79.227.15:4876 , piece_id=4DXEBIOQ2FUILNPHIB46LKFM4YLMH6BAIT2DFIGCKERBGJC7FUTA, seg_pos={0 0}, upload=470044]: Uploading piece... Dec 27 23:26:10.012502837 (535h57m9.883585892s) [noise=true , address=145.79.227.3:4150 , piece_id=M234YSLS3IVIZ2OHJOJ32CHT4VXNF34I5L24HEG2PM6M3ZLW3A2A, seg_pos={0 0}, upload=470044]: Uploading piece... Dec 27 23:26:10.013975953 (535h57m9.885058988s) [noise=true , address=145.79.227.17:4970 , piece_id=KVVNFZ5GLULOHABBOWRA2D7UHDEO2JIVL43I5X26BMJCH2YKDMZA, seg_pos={0 0}, upload=470044]: Uploading piece... Dec 27 23:26:10.015358905 (535h57m9.88644195s) [noise=true , address=145.79.227.34:4907 , piece_id=BS22A76AGBBM4J336LX7L55U2GINLCOA3K4GEIZ524ULQ7FGH5EQ, seg_pos={0 0}, upload=470044]: Uploading piece... Dec 27 23:26:10.016702564 (535h57m9.887785609s) [noise=true , address=45.91.116.12:4225 , piece_id=VVGUEG6RID7L5VLY234AMCZC3JQAL6RPG4QRVT7LG3CYHT6WFFIQ, seg_pos={0 0}, upload=470044]: Uploading piece... Dec 27 23:26:11.689018852 (535h57m11.560101887s) [upload=470044]: Next returned nil segment Dec 27 23:26:11.689040453 (535h57m11.560123498s) [upload=470044]: Waiting for error group managing segments... Dec 27 23:26:12.000885878 (535h57m11.871968933s) [noise=true , address=145.79.227.37:4565 , piece_id=SMAAEQIVIAKKWWZVFJF3BXJVKRNBAEMJOKKXRJVEETKPV2RM3WHA, seg_pos={0 0}, upload=470044]: Done uploading piece. err: <nil> Dec 27 23:26:12.004817334 (535h57m11.875900369s) [noise=true , address=145.79.227.11:4731 , piece_id=JI2OU4IDEBRUYAKDKWFVCMB7F4ZVE7XHZDA6OGTLSTGGGNRLRYUA, seg_pos={0 0}, upload=470044]: Done uploading piece. err: <nil> Dec 27 23:26:12.021315810 (535h57m11.892398855s) [noise=true , address=145.79.227.2:4114 , piece_id=VP5LKMP3SI6ACXAJRE67AZVX7QSZIYSEHMOB4F4MOYMITNUTMUOQ, seg_pos={0 0}, upload=470044]: Done uploading piece. err: <nil> Dec 27 23:26:12.026435118 (535h57m11.897518163s) [noise=true , address=145.79.227.39:4015 , piece_id=SCNEGZ2PEZKRZCZG37G4GZEXJPQ5BNIVLRKJBLLOMDJUTGSUVNUQ, seg_pos={0 0}, upload=470044]: Done uploading piece. err: <nil> Dec 27 23:26:12.026959316 (535h57m11.898042351s) [noise=true , address=145.79.227.3:4150 , piece_id=M234YSLS3IVIZ2OHJOJ32CHT4VXNF34I5L24HEG2PM6M3ZLW3A2A, seg_pos={0 0}, upload=470044]: Done uploading piece. err: <nil> Dec 27 23:26:12.031557762 (535h57m11.902640807s) [noise=true , address=145.79.227.36:4518 , piece_id=5Y24USRES4CGVFJLIKQMEN6B47PTLVGH7KTGFOBAHH2P4INIQIBA, seg_pos={0 0}, upload=470044]: Done uploading piece. err: <nil> Dec 27 23:26:12.033033843 (535h57m11.904116888s) [noise=true , address=145.79.227.15:4876 , piece_id=4DXEBIOQ2FUILNPHIB46LKFM4YLMH6BAIT2DFIGCKERBGJC7FUTA, seg_pos={0 0}, upload=470044]: Done uploading piece. err: <nil> Dec 27 23:26:12.035046155 (535h57m11.90612919s) [noise=true , address=145.79.227.17:4970 , piece_id=KVVNFZ5GLULOHABBOWRA2D7UHDEO2JIVL43I5X26BMJCH2YKDMZA, seg_pos={0 0}, upload=470044]: Done uploading piece. err: <nil> Dec 27 23:26:12.166553877 (535h57m12.037636912s) [noise=true , address=77.90.31.10:4520 , piece_id=YU6FIY5TJBTVS5673X66XUKXX3KGWELIWOG4IALNZK2Z2XG6MPYA, seg_pos={0 0}, upload=470044]: Done uploading piece. err: <nil> Dec 27 23:26:12.240454325 (535h57m12.111537361s) [noise=true , address=45.91.116.5:4297 , piece_id=WAUKVMZTOMP6SCUH4TKPJMTMNAMCCYXU54VUWHOA3SKRGJ2OOPSQ, seg_pos={0 0}, upload=470044]: Done uploading piece. err: <nil> Dec 27 23:26:12.247555477 (535h57m12.118638512s) [noise=true , address=45.91.116.10:4083 , piece_id=JW5F6KSZ7F6HTRDYJPRNJE2S7EJSXI66CKTH7JULYWHPNB3EF5MA, seg_pos={0 0}, upload=470044]: Done uploading piece. err: <nil> Dec 27 23:26:12.316735448 (535h57m12.187818493s) [noise=true , address=45.91.116.14:4189 , piece_id=XZVQTGPHBUH243NCG67L33UFBN65DWKNBBL5QRU3RKHPR7ORNWVQ, seg_pos={0 0}, upload=470044]: Done uploading piece. err: <nil> Dec 27 23:26:12.323218121 (535h57m12.194301167s) [noise=true , address=45.91.116.8:4047 , piece_id=KICSFLOMS4GPZGGADAGVRVUVYFRYACRHAYFKHPZKMLQVCUU3ZFSA, seg_pos={0 0}, upload=470044]: Done uploading piece. err: <nil> Dec 27 23:26:12.323241977 (535h57m12.194325012s) [seg_pos={0 0}, upload=470044]: Segment reached optimal threshold of 13 pieces. Dec 27 23:26:12.323615869 (535h57m12.194698915s) [noise=true , address=45.91.116.11:4118 , piece_id=ZCYN7MKSCJ4TH3OALKIASCCW7AIGJO76NNAFT3R75EPWROMIDTQQ, seg_pos={0 0}, upload=470044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:47660->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:47660->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47660->45.91.116.11:4118: use of closed network connection Dec 27 23:26:12.323625568 (535h57m12.194708613s) [noise=true , address=45.91.116.11:4118 , piece_id=ZCYN7MKSCJ4TH3OALKIASCCW7AIGJO76NNAFT3R75EPWROMIDTQQ, seg_pos={0 0}, upload=470044]: Long tail cancellation due to optimal threshold Dec 27 23:26:12.323776735 (535h57m12.19485978s) [noise=true , address=45.91.116.12:4225 , piece_id=VVGUEG6RID7L5VLY234AMCZC3JQAL6RPG4QRVT7LG3CYHT6WFFIQ, seg_pos={0 0}, upload=470044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 27 23:26:12.323851988 (535h57m12.194935043s) [noise=true , address=45.91.116.12:4225 , piece_id=VVGUEG6RID7L5VLY234AMCZC3JQAL6RPG4QRVT7LG3CYHT6WFFIQ, seg_pos={0 0}, upload=470044]: Long tail cancellation due to optimal threshold Dec 27 23:26:12.324046219 (535h57m12.195129254s) [noise=true , address=145.79.227.34:4907 , piece_id=BS22A76AGBBM4J336LX7L55U2GINLCOA3K4GEIZ524ULQ7FGH5EQ, seg_pos={0 0}, upload=470044]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 23:26:12.324057540 (535h57m12.195140575s) [noise=true , address=145.79.227.34:4907 , piece_id=BS22A76AGBBM4J336LX7L55U2GINLCOA3K4GEIZ524ULQ7FGH5EQ, seg_pos={0 0}, upload=470044]: Long tail cancellation due to optimal threshold Dec 27 23:26:12.324111002 (535h57m12.195194037s) [seg_pos={0 0}, upload=470044]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:26:12.324223105 (535h57m12.19530614s) [upload=470044]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:26:12.324229217 (535h57m12.195312262s) [upload=470044]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:26:12.620955924 (535h57m12.492038969s) [upload=470044]: Done uploading segments... Dec 27 23:26:12.620969229 (535h57m12.492052264s) [upload=470044]: Upload finished. err: <nil> Duration: 30.00330642s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.559394346s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:26:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.559423553s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 3.583419542s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:26:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.58344532s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.569836086s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:25:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.569862858s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [X....X.......X.....X..........................XXX.....XX.XX.] duration: 24.441029992s trace: https://jaeger.dev.storj.io/trace/25528e2330afb65d FAIL (2025-12-27 23:25:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 54TWPEUFYAD6YO5FC3NB2R77EWQVX76VUULQQ4ASSJIMVWN4OJMQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: K67Z53PAEXWLDJTL54QBUPHOLXIAVSG6LNZPM24HOUT4E754KJCA) context deadline exceeded ======LOGS====== Dec 27 23:24:05.942343608 (535h55m5.813426643s) [upload=470014]: Starting upload Dec 27 23:24:05.942410355 (535h55m5.81349339s) [upload=470014]: Done starting upload Dec 27 23:24:05.942440022 (535h55m5.813523057s) [upload=470014]: Uploading segments... Dec 27 23:24:05.942606308 (535h55m5.813689343s) [upload=470014]: Got next segment. Inline: false Dec 27 23:24:05.942671423 (535h55m5.813754458s) [upload=470014]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:24:05.942678055 (535h55m5.8137611s) [upload=470014]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:24:06.186067679 (535h55m6.057150724s) [seg_pos={0 0}, upload=470014]: Begin upload segment... Dec 27 23:24:06.186248804 (535h55m6.057331849s) [seg_pos={0 0}, upload=470014]: Done begin upload segment. Dec 27 23:24:06.186308969 (535h55m6.057392014s) [noise=true , address=145.79.227.9:4365 , piece_id=RHI6UKMG5LGBFB2JTV5O7YOUL4T7XT4NVZ2YH6FMKRG7OYEH3SKA, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.186325710 (535h55m6.057408745s) [noise=true , address=77.90.31.17:4773 , piece_id=4TQ27GQBBG7ZVJQYZBMGQQ2MRXA6TMNC2CCPYL7SYMQW5ZN7YCWQ, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.186942996 (535h55m6.058026031s) [noise=true , address=145.79.227.51:4410 , piece_id=5FMIAQLTJHARRFCR4P2DU4LHF2LCJUMYEDS5JWOK7NXXHB7WFDWA, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.187387663 (535h55m6.058470708s) [noise=true , address=45.91.116.6:4974 , piece_id=VJEG5TWGPGBRGNU2ESLRYLLS6FUMVL5EROLEONWZ744QTQJNF5KQ, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.187758479 (535h55m6.058841514s) [noise=true , address=45.91.116.9:4262 , piece_id=VAY4WHC2DMZB2UBFOFZHBVUERDAB2YMWJVY4RX3WN5FKSC47UI4A, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.188128363 (535h55m6.059211408s) [noise=true , address=145.79.227.47:4266 , piece_id=OGHACFB2AXDAVJ6W2WTOHYMO6VVMARS3IKTOFM6VHLN5QOO6L5OA, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.188597797 (535h55m6.059680842s) [noise=true , address=45.91.116.5:4299 , piece_id=54TWPEUFYAD6YO5FC3NB2R77EWQVX76VUULQQ4ASSJIMVWN4OJMQ, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.189143275 (535h55m6.06022632s) [noise=true , address=145.79.227.14:4839 , piece_id=M2VB5XWDJTPQGCJJ6OMCSUNQ2TH6IATDJ3A4QHG4TQMLZQAA47RQ, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.189669146 (535h55m6.060752191s) [noise=true , address=145.79.227.15:4874 , piece_id=QKMEZJVDKNJEVVCQUOD5P4WV3JLTWR4K5BPVJOGM4QW56CYHA2XA, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.191149716 (535h55m6.062232761s) [noise=true , address=77.90.31.15:4697 , piece_id=DFOKH7KHXM4PMZIGJ4O5KGJJMMKUTTP5RLK5LATDIUUT2OCS4ZXA, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.191229548 (535h55m6.062312603s) [noise=true , address=145.79.227.52:4447 , piece_id=JAFDIZALAKPNI4PBFZS43YL3HS7MTK3A2FEEHFHZX5ZGLLRGLWEQ, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.191816365 (535h55m6.0628994s) [noise=true , address=77.90.31.13:4627 , piece_id=SYO2VR73WEKPHS55SMVURTMNU6OO34WTXKVHKF3DIT2MLGC2U6BQ, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.193876438 (535h55m6.064959483s) [noise=true , address=45.91.116.8:4045 , piece_id=K67Z53PAEXWLDJTL54QBUPHOLXIAVSG6LNZPM24HOUT4E754KJCA, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.198170265 (535h55m6.0692533s) [noise=true , address=145.79.227.38:4978 , piece_id=2NLBSGX6BKUFLT5DM4HSJOVGD7VGCPTRBHBVB5QXD57EUIXJGGKA, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.200032009 (535h55m6.071115044s) [noise=true , address=145.79.227.2:4114 , piece_id=YVMF6VEXPFQYTSPE5KI635CG5W4WBVEO6ATTLK536HUA6LAZ2UDA, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.201652546 (535h55m6.072735591s) [noise=true , address=145.79.227.34:4907 , piece_id=L7SMAXHTPQYEEAS3U625DTM7LVP4Q6W47H6RMB67IU4JG6YGN63Q, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:07.895427413 (535h55m7.766510448s) [upload=470014]: Next returned nil segment Dec 27 23:24:07.895443793 (535h55m7.766526839s) [upload=470014]: Waiting for error group managing segments... Dec 27 23:24:08.181456779 (535h55m8.052539814s) [noise=true , address=145.79.227.52:4447 , piece_id=JAFDIZALAKPNI4PBFZS43YL3HS7MTK3A2FEEHFHZX5ZGLLRGLWEQ, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.191568834 (535h55m8.062651879s) [noise=true , address=145.79.227.47:4266 , piece_id=OGHACFB2AXDAVJ6W2WTOHYMO6VVMARS3IKTOFM6VHLN5QOO6L5OA, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.195325067 (535h55m8.066408102s) [noise=true , address=145.79.227.38:4978 , piece_id=2NLBSGX6BKUFLT5DM4HSJOVGD7VGCPTRBHBVB5QXD57EUIXJGGKA, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.208645622 (535h55m8.079728657s) [noise=true , address=145.79.227.2:4114 , piece_id=YVMF6VEXPFQYTSPE5KI635CG5W4WBVEO6ATTLK536HUA6LAZ2UDA, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.209394417 (535h55m8.080477452s) [noise=true , address=145.79.227.15:4874 , piece_id=QKMEZJVDKNJEVVCQUOD5P4WV3JLTWR4K5BPVJOGM4QW56CYHA2XA, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.210161457 (535h55m8.081244492s) [noise=true , address=145.79.227.14:4839 , piece_id=M2VB5XWDJTPQGCJJ6OMCSUNQ2TH6IATDJ3A4QHG4TQMLZQAA47RQ, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.226489699 (535h55m8.097572734s) [noise=true , address=145.79.227.9:4365 , piece_id=RHI6UKMG5LGBFB2JTV5O7YOUL4T7XT4NVZ2YH6FMKRG7OYEH3SKA, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.370627602 (535h55m8.241710637s) [noise=true , address=77.90.31.17:4773 , piece_id=4TQ27GQBBG7ZVJQYZBMGQQ2MRXA6TMNC2CCPYL7SYMQW5ZN7YCWQ, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.381309773 (535h55m8.252392818s) [noise=true , address=77.90.31.13:4627 , piece_id=SYO2VR73WEKPHS55SMVURTMNU6OO34WTXKVHKF3DIT2MLGC2U6BQ, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.431431833 (535h55m8.302514868s) [noise=true , address=45.91.116.8:4045 , piece_id=K67Z53PAEXWLDJTL54QBUPHOLXIAVSG6LNZPM24HOUT4E754KJCA, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.535989378 (535h55m8.407072413s) [noise=true , address=45.91.116.9:4262 , piece_id=VAY4WHC2DMZB2UBFOFZHBVUERDAB2YMWJVY4RX3WN5FKSC47UI4A, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:09.182773011 (535h55m9.053856056s) [noise=true , address=45.91.116.6:4974 , piece_id=VJEG5TWGPGBRGNU2ESLRYLLS6FUMVL5EROLEONWZ744QTQJNF5KQ, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:09.218043492 (535h55m9.089126527s) [noise=true , address=45.91.116.5:4299 , piece_id=54TWPEUFYAD6YO5FC3NB2R77EWQVX76VUULQQ4ASSJIMVWN4OJMQ, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:09.218063299 (535h55m9.089146334s) [seg_pos={0 0}, upload=470014]: Segment reached optimal threshold of 13 pieces. Dec 27 23:24:09.218285733 (535h55m9.089368778s) [noise=true , address=145.79.227.34:4907 , piece_id=L7SMAXHTPQYEEAS3U625DTM7LVP4Q6W47H6RMB67IU4JG6YGN63Q, seg_pos={0 0}, upload=470014]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 23:24:09.218292646 (535h55m9.089375681s) [noise=true , address=145.79.227.34:4907 , piece_id=L7SMAXHTPQYEEAS3U625DTM7LVP4Q6W47H6RMB67IU4JG6YGN63Q, seg_pos={0 0}, upload=470014]: Long tail cancellation due to optimal threshold Dec 27 23:24:09.218491575 (535h55m9.08957462s) [noise=true , address=77.90.31.15:4697 , piece_id=DFOKH7KHXM4PMZIGJ4O5KGJJMMKUTTP5RLK5LATDIUUT2OCS4ZXA, seg_pos={0 0}, upload=470014]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:41580->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:41580->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41580->77.90.31.15:4697: use of closed network connection Dec 27 23:24:09.218497907 (535h55m9.089580942s) [noise=true , address=77.90.31.15:4697 , piece_id=DFOKH7KHXM4PMZIGJ4O5KGJJMMKUTTP5RLK5LATDIUUT2OCS4ZXA, seg_pos={0 0}, upload=470014]: Long tail cancellation due to optimal threshold Dec 27 23:24:09.218627865 (535h55m9.08971095s) [noise=true , address=145.79.227.51:4410 , piece_id=5FMIAQLTJHARRFCR4P2DU4LHF2LCJUMYEDS5JWOK7NXXHB7WFDWA, seg_pos={0 0}, upload=470014]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 27 23:24:09.218655827 (535h55m9.089738902s) [noise=true , address=145.79.227.51:4410 , piece_id=5FMIAQLTJHARRFCR4P2DU4LHF2LCJUMYEDS5JWOK7NXXHB7WFDWA, seg_pos={0 0}, upload=470014]: Long tail cancellation due to optimal threshold Dec 27 23:24:09.218687237 (535h55m9.089770272s) [seg_pos={0 0}, upload=470014]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:24:09.218869845 (535h55m9.08995288s) [upload=470014]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:24:09.218902126 (535h55m9.089985191s) [upload=470014]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:24:09.509377239 (535h55m9.380460274s) [upload=470014]: Done uploading segments... Dec 27 23:24:09.509395755 (535h55m9.38047879s) [upload=470014]: Upload finished. err: <nil> Duration: 24.441058617s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 4.217441555s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:24:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.217467746s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.X....X.......X.....X..........................XXX.....XX.XX] duration: 30.002863054s trace: https://jaeger.dev.storj.io/trace/25528e2330afb65d FAIL (2025-12-27 23:24:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: 54TWPEUFYAD6YO5FC3NB2R77EWQVX76VUULQQ4ASSJIMVWN4OJMQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: K67Z53PAEXWLDJTL54QBUPHOLXIAVSG6LNZPM24HOUT4E754KJCA) context deadline exceeded ======LOGS====== Dec 27 23:24:05.942343608 (535h55m5.813426643s) [upload=470014]: Starting upload Dec 27 23:24:05.942410355 (535h55m5.81349339s) [upload=470014]: Done starting upload Dec 27 23:24:05.942440022 (535h55m5.813523057s) [upload=470014]: Uploading segments... Dec 27 23:24:05.942606308 (535h55m5.813689343s) [upload=470014]: Got next segment. Inline: false Dec 27 23:24:05.942671423 (535h55m5.813754458s) [upload=470014]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:24:05.942678055 (535h55m5.8137611s) [upload=470014]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:24:06.186067679 (535h55m6.057150724s) [seg_pos={0 0}, upload=470014]: Begin upload segment... Dec 27 23:24:06.186248804 (535h55m6.057331849s) [seg_pos={0 0}, upload=470014]: Done begin upload segment. Dec 27 23:24:06.186308969 (535h55m6.057392014s) [noise=true , address=145.79.227.9:4365 , piece_id=RHI6UKMG5LGBFB2JTV5O7YOUL4T7XT4NVZ2YH6FMKRG7OYEH3SKA, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.186325710 (535h55m6.057408745s) [noise=true , address=77.90.31.17:4773 , piece_id=4TQ27GQBBG7ZVJQYZBMGQQ2MRXA6TMNC2CCPYL7SYMQW5ZN7YCWQ, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.186942996 (535h55m6.058026031s) [noise=true , address=145.79.227.51:4410 , piece_id=5FMIAQLTJHARRFCR4P2DU4LHF2LCJUMYEDS5JWOK7NXXHB7WFDWA, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.187387663 (535h55m6.058470708s) [noise=true , address=45.91.116.6:4974 , piece_id=VJEG5TWGPGBRGNU2ESLRYLLS6FUMVL5EROLEONWZ744QTQJNF5KQ, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.187758479 (535h55m6.058841514s) [noise=true , address=45.91.116.9:4262 , piece_id=VAY4WHC2DMZB2UBFOFZHBVUERDAB2YMWJVY4RX3WN5FKSC47UI4A, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.188128363 (535h55m6.059211408s) [noise=true , address=145.79.227.47:4266 , piece_id=OGHACFB2AXDAVJ6W2WTOHYMO6VVMARS3IKTOFM6VHLN5QOO6L5OA, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.188597797 (535h55m6.059680842s) [noise=true , address=45.91.116.5:4299 , piece_id=54TWPEUFYAD6YO5FC3NB2R77EWQVX76VUULQQ4ASSJIMVWN4OJMQ, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.189143275 (535h55m6.06022632s) [noise=true , address=145.79.227.14:4839 , piece_id=M2VB5XWDJTPQGCJJ6OMCSUNQ2TH6IATDJ3A4QHG4TQMLZQAA47RQ, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.189669146 (535h55m6.060752191s) [noise=true , address=145.79.227.15:4874 , piece_id=QKMEZJVDKNJEVVCQUOD5P4WV3JLTWR4K5BPVJOGM4QW56CYHA2XA, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.191149716 (535h55m6.062232761s) [noise=true , address=77.90.31.15:4697 , piece_id=DFOKH7KHXM4PMZIGJ4O5KGJJMMKUTTP5RLK5LATDIUUT2OCS4ZXA, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.191229548 (535h55m6.062312603s) [noise=true , address=145.79.227.52:4447 , piece_id=JAFDIZALAKPNI4PBFZS43YL3HS7MTK3A2FEEHFHZX5ZGLLRGLWEQ, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.191816365 (535h55m6.0628994s) [noise=true , address=77.90.31.13:4627 , piece_id=SYO2VR73WEKPHS55SMVURTMNU6OO34WTXKVHKF3DIT2MLGC2U6BQ, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.193876438 (535h55m6.064959483s) [noise=true , address=45.91.116.8:4045 , piece_id=K67Z53PAEXWLDJTL54QBUPHOLXIAVSG6LNZPM24HOUT4E754KJCA, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.198170265 (535h55m6.0692533s) [noise=true , address=145.79.227.38:4978 , piece_id=2NLBSGX6BKUFLT5DM4HSJOVGD7VGCPTRBHBVB5QXD57EUIXJGGKA, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.200032009 (535h55m6.071115044s) [noise=true , address=145.79.227.2:4114 , piece_id=YVMF6VEXPFQYTSPE5KI635CG5W4WBVEO6ATTLK536HUA6LAZ2UDA, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:06.201652546 (535h55m6.072735591s) [noise=true , address=145.79.227.34:4907 , piece_id=L7SMAXHTPQYEEAS3U625DTM7LVP4Q6W47H6RMB67IU4JG6YGN63Q, seg_pos={0 0}, upload=470014]: Uploading piece... Dec 27 23:24:07.895427413 (535h55m7.766510448s) [upload=470014]: Next returned nil segment Dec 27 23:24:07.895443793 (535h55m7.766526839s) [upload=470014]: Waiting for error group managing segments... Dec 27 23:24:08.181456779 (535h55m8.052539814s) [noise=true , address=145.79.227.52:4447 , piece_id=JAFDIZALAKPNI4PBFZS43YL3HS7MTK3A2FEEHFHZX5ZGLLRGLWEQ, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.191568834 (535h55m8.062651879s) [noise=true , address=145.79.227.47:4266 , piece_id=OGHACFB2AXDAVJ6W2WTOHYMO6VVMARS3IKTOFM6VHLN5QOO6L5OA, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.195325067 (535h55m8.066408102s) [noise=true , address=145.79.227.38:4978 , piece_id=2NLBSGX6BKUFLT5DM4HSJOVGD7VGCPTRBHBVB5QXD57EUIXJGGKA, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.208645622 (535h55m8.079728657s) [noise=true , address=145.79.227.2:4114 , piece_id=YVMF6VEXPFQYTSPE5KI635CG5W4WBVEO6ATTLK536HUA6LAZ2UDA, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.209394417 (535h55m8.080477452s) [noise=true , address=145.79.227.15:4874 , piece_id=QKMEZJVDKNJEVVCQUOD5P4WV3JLTWR4K5BPVJOGM4QW56CYHA2XA, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.210161457 (535h55m8.081244492s) [noise=true , address=145.79.227.14:4839 , piece_id=M2VB5XWDJTPQGCJJ6OMCSUNQ2TH6IATDJ3A4QHG4TQMLZQAA47RQ, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.226489699 (535h55m8.097572734s) [noise=true , address=145.79.227.9:4365 , piece_id=RHI6UKMG5LGBFB2JTV5O7YOUL4T7XT4NVZ2YH6FMKRG7OYEH3SKA, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.370627602 (535h55m8.241710637s) [noise=true , address=77.90.31.17:4773 , piece_id=4TQ27GQBBG7ZVJQYZBMGQQ2MRXA6TMNC2CCPYL7SYMQW5ZN7YCWQ, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.381309773 (535h55m8.252392818s) [noise=true , address=77.90.31.13:4627 , piece_id=SYO2VR73WEKPHS55SMVURTMNU6OO34WTXKVHKF3DIT2MLGC2U6BQ, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.431431833 (535h55m8.302514868s) [noise=true , address=45.91.116.8:4045 , piece_id=K67Z53PAEXWLDJTL54QBUPHOLXIAVSG6LNZPM24HOUT4E754KJCA, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:08.535989378 (535h55m8.407072413s) [noise=true , address=45.91.116.9:4262 , piece_id=VAY4WHC2DMZB2UBFOFZHBVUERDAB2YMWJVY4RX3WN5FKSC47UI4A, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:09.182773011 (535h55m9.053856056s) [noise=true , address=45.91.116.6:4974 , piece_id=VJEG5TWGPGBRGNU2ESLRYLLS6FUMVL5EROLEONWZ744QTQJNF5KQ, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:09.218043492 (535h55m9.089126527s) [noise=true , address=45.91.116.5:4299 , piece_id=54TWPEUFYAD6YO5FC3NB2R77EWQVX76VUULQQ4ASSJIMVWN4OJMQ, seg_pos={0 0}, upload=470014]: Done uploading piece. err: <nil> Dec 27 23:24:09.218063299 (535h55m9.089146334s) [seg_pos={0 0}, upload=470014]: Segment reached optimal threshold of 13 pieces. Dec 27 23:24:09.218285733 (535h55m9.089368778s) [noise=true , address=145.79.227.34:4907 , piece_id=L7SMAXHTPQYEEAS3U625DTM7LVP4Q6W47H6RMB67IU4JG6YGN63Q, seg_pos={0 0}, upload=470014]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 23:24:09.218292646 (535h55m9.089375681s) [noise=true , address=145.79.227.34:4907 , piece_id=L7SMAXHTPQYEEAS3U625DTM7LVP4Q6W47H6RMB67IU4JG6YGN63Q, seg_pos={0 0}, upload=470014]: Long tail cancellation due to optimal threshold Dec 27 23:24:09.218491575 (535h55m9.08957462s) [noise=true , address=77.90.31.15:4697 , piece_id=DFOKH7KHXM4PMZIGJ4O5KGJJMMKUTTP5RLK5LATDIUUT2OCS4ZXA, seg_pos={0 0}, upload=470014]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:41580->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:41580->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41580->77.90.31.15:4697: use of closed network connection Dec 27 23:24:09.218497907 (535h55m9.089580942s) [noise=true , address=77.90.31.15:4697 , piece_id=DFOKH7KHXM4PMZIGJ4O5KGJJMMKUTTP5RLK5LATDIUUT2OCS4ZXA, seg_pos={0 0}, upload=470014]: Long tail cancellation due to optimal threshold Dec 27 23:24:09.218627865 (535h55m9.08971095s) [noise=true , address=145.79.227.51:4410 , piece_id=5FMIAQLTJHARRFCR4P2DU4LHF2LCJUMYEDS5JWOK7NXXHB7WFDWA, seg_pos={0 0}, upload=470014]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX): protocol: expected piece hash; context canceled; EOF Dec 27 23:24:09.218655827 (535h55m9.089738902s) [noise=true , address=145.79.227.51:4410 , piece_id=5FMIAQLTJHARRFCR4P2DU4LHF2LCJUMYEDS5JWOK7NXXHB7WFDWA, seg_pos={0 0}, upload=470014]: Long tail cancellation due to optimal threshold Dec 27 23:24:09.218687237 (535h55m9.089770272s) [seg_pos={0 0}, upload=470014]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:24:09.218869845 (535h55m9.08995288s) [upload=470014]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:24:09.218902126 (535h55m9.089985191s) [upload=470014]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:24:09.509377239 (535h55m9.380460274s) [upload=470014]: Done uploading segments... Dec 27 23:24:09.509395755 (535h55m9.38047879s) [upload=470014]: Upload finished. err: <nil> Duration: 30.00288655s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.704131346s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:23:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.704467456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..X....X.......X.....X..........................XXX.....XX.X] duration: 30.003691057s trace: https://jaeger.dev.storj.io/trace/7ed13e04c870ecf2 FAIL (2025-12-27 23:23:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 23:23:09.593773051 (535h54m9.464856096s) [upload=470003]: Starting upload Dec 27 23:23:09.593884253 (535h54m9.464967298s) [upload=470003]: Done starting upload Dec 27 23:23:09.593987119 (535h54m9.465070184s) [upload=470003]: Uploading segments... Dec 27 23:23:09.594185156 (535h54m9.465268191s) [upload=470003]: Got next segment. Inline: false Dec 27 23:23:09.594248766 (535h54m9.465331801s) [upload=470003]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:23:09.594255680 (535h54m9.465338715s) [upload=470003]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:23:09.837618042 (535h54m9.708701077s) [seg_pos={0 0}, upload=470003]: Begin upload segment... Dec 27 23:23:09.837662827 (535h54m9.708745862s) [seg_pos={0 0}, upload=470003]: Done begin upload segment. Dec 27 23:23:09.837701711 (535h54m9.708784756s) [noise=true , address=77.90.31.12:4590 , piece_id=OA6S2ESYDXRVVVPAAUHNXZGZWCARWYRESQ76MKO2WHT55GJUT5TA, seg_pos={0 0}, upload=470003]: Uploading piece... Dec 27 23:23:09.837876404 (535h54m9.708959449s) [noise=true , address=77.90.31.19:4844 , piece_id=5ABRZI7YNIYCP7XRAAGKVTOZNJ6KHBJ5UACZ6CTWISUL7MDKZECQ, seg_pos={0 0}, upload=470003]: Uploading piece... Dec 27 23:23:09.838602957 (535h54m9.709686002s) [noise=true , address=45.91.116.11:4119 , piece_id=DLZ6IUAQ5OQZCMINUOB46FIORGO4PTXXQI6W4ZUDXH2YUARKECMQ, seg_pos={0 0}, upload=470003]: Uploading piece... Dec 27 23:23:09.839034218 (535h54m9.710117263s) [noise=true , address=77.90.31.13:4627 , piece_id=AUDXJBYMNJLZWSVHAYS4XUP4RB6AS4GDZPOQFDOB46CLPVXOTIBQ, seg_pos={0 0}, upload=470003]: Uploading piece... Dec 27 23:23:09.839420544 (535h54m9.710503589s) [noise=true , address=145.79.227.46:4601 , piece_id=Z24KA4PZU7SEMV35H3AUZ5XIHO6BZKCNKDEBBTBOZGOR5RW6HBKA, seg_pos={0 0}, upload=470003]: Uploading piece... Dec 27 23:23:09.839823301 (535h54m9.710906346s) [noise=true , address=45.91.116.5:4300 , piece_id=ME4KLXS7JR4MNVJWHT6ETBLSWOWUSI6U77BMNZLNL7YYF74LHH5A, seg_pos={0 0}, upload=470003]: Uploading piece... Dec 27 23:23:09.839827920 (535h54m9.710910955s) [noise=true , address=145.79.227.15:4874 , piece_id=J3PHR62MKFPIR76Q4MW3KUN7DVLX6VHOYF2WP6GR4GLJOWORHG3A, seg_pos={0 0}, upload=470003]: Uploading piece... Dec 27 23:23:09.840112011 (535h54m9.711195046s) [noise=true , address=145.79.227.4:4186 , piece_id=GVCPJUZOHKR4T4BM4AIKMLL7ORU6HQHZQUK2LHMC6O6NTQXKTLOA, seg_pos={0 0}, upload=470003]: Uploading piece... Dec 27 23:23:09.840224454 (535h54m9.711307489s) [noise=true , address=145.79.227.37:4566 , piece_id=SFGJRLWM7KJT4UKQCPKUTYE2LDOTUPGVIA5ZVMU3GGCY2O7BXVKA, seg_pos={0 0}, upload=470003]: Uploading piece... Dec 27 23:23:09.840416781 (535h54m9.711499826s) [noise=true , address=45.91.116.7:4012 , piece_id=3BJXFVEBRZKKAIIKXGP44RKPTUMLEKJNZYLHZGQEPXSUAJQS2V6A, seg_pos={0 0}, upload=470003]: Uploading piece... Dec 27 23:23:09.840713856 (535h54m9.711796901s) [noise=true , address=145.79.227.43:4159 , piece_id=E3KSWPVWA2BJDD2LOYS4XTBGG5M27I6KHJPYDAYSLS6P5IKLSLXA, seg_pos={0 0}, upload=470003]: Uploading piece... Dec 27 23:23:09.841014219 (535h54m9.712097254s) [noise=true , address=145.79.227.16:4911 , piece_id=EBMC5OMHWSTQTV6243KOHWG3L6FNFO5TE2BYCIXYL2NHV3GXBU3Q, seg_pos={0 0}, upload=470003]: Uploading piece... Dec 27 23:23:09.841343226 (535h54m9.712426261s) [noise=true , address=145.79.227.11:4730 , piece_id=T5QM3B4A25RPQUSWPK3KRKO6YRHCWJCG4NCR63ED3UKFDWPOEPVA, seg_pos={0 0}, upload=470003]: Uploading piece... Dec 27 23:23:09.841647023 (535h54m9.712730058s) [noise=true , address=145.79.227.9:4365 , piece_id=AA5PZ4OUELU6UD5QVBRPDIXXSLE6LIZP5VT3DQC4D5UUUYRU7PHQ, seg_pos={0 0}, upload=470003]: Uploading piece... Dec 27 23:23:09.841994735 (535h54m9.713077771s) [noise=true , address=77.90.31.21:4878 , piece_id=YAX4SL25P7L53DJAR6J4U7HUS3RAXMY4R5QFUZA6NJADBHEFJFYA, seg_pos={0 0}, upload=470003]: Uploading piece... Dec 27 23:23:09.842414074 (535h54m9.713497109s) [noise=true , address=145.79.227.44:4193 , piece_id=6WKCDJJQMLQKTAJQR2DBZIHSSFJVP77VXQJZTYMRAT7XUUYGA6PA, seg_pos={0 0}, upload=470003]: Uploading piece... Dec 27 23:23:11.547407613 (535h54m11.418490648s) [upload=470003]: Next returned nil segment Dec 27 23:23:11.547424406 (535h54m11.418507441s) [upload=470003]: Waiting for error group managing segments... Dec 27 23:23:11.830848261 (535h54m11.701931306s) [noise=true , address=145.79.227.9:4365 , piece_id=AA5PZ4OUELU6UD5QVBRPDIXXSLE6LIZP5VT3DQC4D5UUUYRU7PHQ, seg_pos={0 0}, upload=470003]: Done uploading piece. err: <nil> Dec 27 23:23:11.846863126 (535h54m11.717946171s) [noise=true , address=145.79.227.16:4911 , piece_id=EBMC5OMHWSTQTV6243KOHWG3L6FNFO5TE2BYCIXYL2NHV3GXBU3Q, seg_pos={0 0}, upload=470003]: Done uploading piece. err: <nil> Dec 27 23:23:11.850844497 (535h54m11.721927533s) [noise=true , address=145.79.227.43:4159 , piece_id=E3KSWPVWA2BJDD2LOYS4XTBGG5M27I6KHJPYDAYSLS6P5IKLSLXA, seg_pos={0 0}, upload=470003]: Done uploading piece. err: <nil> Dec 27 23:23:11.851887654 (535h54m11.722970689s) [noise=true , address=145.79.227.11:4730 , piece_id=T5QM3B4A25RPQUSWPK3KRKO6YRHCWJCG4NCR63ED3UKFDWPOEPVA, seg_pos={0 0}, upload=470003]: Done uploading piece. err: <nil> Dec 27 23:23:11.859706071 (535h54m11.730789106s) [noise=true , address=145.79.227.4:4186 , piece_id=GVCPJUZOHKR4T4BM4AIKMLL7ORU6HQHZQUK2LHMC6O6NTQXKTLOA, seg_pos={0 0}, upload=470003]: Done uploading piece. err: <nil> Dec 27 23:23:11.862516863 (535h54m11.733599898s) [noise=true , address=145.79.227.15:4874 , piece_id=J3PHR62MKFPIR76Q4MW3KUN7DVLX6VHOYF2WP6GR4GLJOWORHG3A, seg_pos={0 0}, upload=470003]: Done uploading piece. err: <nil> Dec 27 23:23:11.865457562 (535h54m11.736540598s) [noise=true , address=145.79.227.37:4566 , piece_id=SFGJRLWM7KJT4UKQCPKUTYE2LDOTUPGVIA5ZVMU3GGCY2O7BXVKA, seg_pos={0 0}, upload=470003]: Done uploading piece. err: <nil> Dec 27 23:23:11.877352984 (535h54m11.748436029s) [noise=true , address=145.79.227.46:4601 , piece_id=Z24KA4PZU7SEMV35H3AUZ5XIHO6BZKCNKDEBBTBOZGOR5RW6HBKA, seg_pos={0 0}, upload=470003]: Done uploading piece. err: <nil> Dec 27 23:23:12.070580220 (535h54m11.941663255s) [noise=true , address=77.90.31.19:4844 , piece_id=5ABRZI7YNIYCP7XRAAGKVTOZNJ6KHBJ5UACZ6CTWISUL7MDKZECQ, seg_pos={0 0}, upload=470003]: Done uploading piece. err: <nil> Dec 27 23:23:12.082324533 (535h54m11.953407568s) [noise=true , address=77.90.31.13:4627 , piece_id=AUDXJBYMNJLZWSVHAYS4XUP4RB6AS4GDZPOQFDOB46CLPVXOTIBQ, seg_pos={0 0}, upload=470003]: Done uploading piece. err: <nil> Dec 27 23:23:12.087980213 (535h54m11.959063248s) [noise=true , address=45.91.116.11:4119 , piece_id=DLZ6IUAQ5OQZCMINUOB46FIORGO4PTXXQI6W4ZUDXH2YUARKECMQ, seg_pos={0 0}, upload=470003]: Done uploading piece. err: <nil> Dec 27 23:23:12.090591315 (535h54m11.96167435s) [noise=true , address=45.91.116.5:4300 , piece_id=ME4KLXS7JR4MNVJWHT6ETBLSWOWUSI6U77BMNZLNL7YYF74LHH5A, seg_pos={0 0}, upload=470003]: Done uploading piece. err: <nil> Dec 27 23:23:12.097279590 (535h54m11.968362635s) [noise=true , address=77.90.31.21:4878 , piece_id=YAX4SL25P7L53DJAR6J4U7HUS3RAXMY4R5QFUZA6NJADBHEFJFYA, seg_pos={0 0}, upload=470003]: Done uploading piece. err: <nil> Dec 27 23:23:12.097311491 (535h54m11.968394536s) [seg_pos={0 0}, upload=470003]: Segment reached optimal threshold of 13 pieces. Dec 27 23:23:12.097547200 (535h54m11.968630245s) [noise=true , address=45.91.116.7:4012 , piece_id=3BJXFVEBRZKKAIIKXGP44RKPTUMLEKJNZYLHZGQEPXSUAJQS2V6A, seg_pos={0 0}, upload=470003]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 27 23:23:12.097557039 (535h54m11.968640084s) [noise=true , address=45.91.116.7:4012 , piece_id=3BJXFVEBRZKKAIIKXGP44RKPTUMLEKJNZYLHZGQEPXSUAJQS2V6A, seg_pos={0 0}, upload=470003]: Long tail cancellation due to optimal threshold Dec 27 23:23:12.097590723 (535h54m11.968673758s) [noise=true , address=77.90.31.12:4590 , piece_id=OA6S2ESYDXRVVVPAAUHNXZGZWCARWYRESQ76MKO2WHT55GJUT5TA, seg_pos={0 0}, upload=470003]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 27 23:23:12.097607935 (535h54m11.96869097s) [noise=true , address=77.90.31.12:4590 , piece_id=OA6S2ESYDXRVVVPAAUHNXZGZWCARWYRESQ76MKO2WHT55GJUT5TA, seg_pos={0 0}, upload=470003]: Long tail cancellation due to optimal threshold Dec 27 23:23:12.097761177 (535h54m11.968844212s) [noise=true , address=145.79.227.44:4193 , piece_id=6WKCDJJQMLQKTAJQR2DBZIHSSFJVP77VXQJZTYMRAT7XUUYGA6PA, seg_pos={0 0}, upload=470003]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 23:23:12.097767299 (535h54m11.968850334s) [noise=true , address=145.79.227.44:4193 , piece_id=6WKCDJJQMLQKTAJQR2DBZIHSSFJVP77VXQJZTYMRAT7XUUYGA6PA, seg_pos={0 0}, upload=470003]: Long tail cancellation due to optimal threshold Dec 27 23:23:12.097784742 (535h54m11.968867777s) [seg_pos={0 0}, upload=470003]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:23:12.097944687 (535h54m11.969027732s) [upload=470003]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:23:12.097950437 (535h54m11.969033482s) [upload=470003]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:23:12.389862154 (535h54m12.260945189s) [upload=470003]: Done uploading segments... Dec 27 23:23:12.389881019 (535h54m12.260964064s) [upload=470003]: Upload finished. err: <nil> Duration: 30.003718028s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.572211499s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:22:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.572234963s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 4.421513839s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:22:21 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.421536874s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X..X....X.......X.....X..........................XXX.....XX.] duration: 7.400465738s trace: https://jaeger.dev.storj.io/trace/191f2761b4f31933 FAIL (2025-12-27 23:22:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 23:21:11.791799688 (535h52m11.662882733s) [upload=469977]: Starting upload Dec 27 23:21:11.791906573 (535h52m11.662989628s) [upload=469977]: Done starting upload Dec 27 23:21:11.791961737 (535h52m11.663044772s) [upload=469977]: Uploading segments... Dec 27 23:21:11.792283730 (535h52m11.663366785s) [upload=469977]: Got next segment. Inline: false Dec 27 23:21:11.792343194 (535h52m11.663426249s) [upload=469977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:21:11.792349646 (535h52m11.663432701s) [upload=469977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:21:12.038302212 (535h52m11.909385257s) [seg_pos={0 0}, upload=469977]: Begin upload segment... Dec 27 23:21:12.038347408 (535h52m11.909430453s) [seg_pos={0 0}, upload=469977]: Done begin upload segment. Dec 27 23:21:12.038370653 (535h52m11.909453688s) [noise=true , address=145.79.227.19:4079 , piece_id=TJNOEVPGNYQ2QK3ZQS2HQTB4WGSWKVRPUGK5GYWPL7OQYPZOLNAA, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.038382235 (535h52m11.90946527s) [noise=true , address=77.90.31.11:4555 , piece_id=KG63ED3NRFLLNSR4AHGCTDH3JXFSWIJTS42LGB4X2PFG63L3TEEA, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.038494097 (535h52m11.909577132s) [noise=true , address=77.90.31.15:4697 , piece_id=34JFX6OGTQNV4NPQGVL5VS76Z5AFWXQAYFQSJFQFWCYAM5DU6WDA, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.038775755 (535h52m11.9098588s) [noise=true , address=145.79.227.33:4869 , piece_id=7QAVC4YREJMSCWN3OBHOH45XYBRTSALY3GLZGSPHQUIZVWBH7GVA, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.039052531 (535h52m11.910135566s) [noise=true , address=45.91.116.14:4189 , piece_id=JH6Z2FA3WMVNIRUUXEDFAVY374PYT6YLZN3ZJLYJBD4JQFPXR7AA, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.039310683 (535h52m11.910393718s) [noise=true , address=77.90.31.12:4590 , piece_id=PK526LNTSML4FVUCJAOF45UNOGB3RCZZ3QJUXG7UKCP4NXTHOEGQ, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.039598931 (535h52m11.910681966s) [noise=true , address=45.91.116.7:4009 , piece_id=M5NUIJTA47GKJSLDTX2H4KN33NVI27CTDOA5MBQJP5JKQ7KQXJGA, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.039908381 (535h52m11.910991416s) [noise=true , address=45.91.116.11:4119 , piece_id=VWOWB6WBMHT7PXSM3DD2DEVFNUCSK2KDIXOX7LXIQW54EL6OW6SQ, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.042145621 (535h52m11.913228676s) [noise=true , address=145.79.227.4:4186 , piece_id=J5VBCZ2UIMKA2JVP7XJLC235WEX2CD6CMQNW6VSQD22G5URYGQFQ, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.043207333 (535h52m11.914290368s) [noise=true , address=145.79.227.32:4836 , piece_id=TQQ2WKMB3D7THEIQMQUZR7P6Y4G3JZHUDODCCF32YUQBSAJQPDAQ, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.044041381 (535h52m11.915124416s) [noise=true , address=145.79.227.15:4876 , piece_id=3NCIXSWK6KFUZLHMZDP7EL7YNIJKMGIKCF7LLKAU3QTQNMSZI3UA, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.047319984 (535h52m11.918403019s) [noise=true , address=145.79.227.16:4911 , piece_id=S757PSGJRTSR3ASTHD7BO6IDJSOXQJSL6Y2E73WN6I4P6Z5DS5BQ, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.049446333 (535h52m11.920529378s) [noise=true , address=45.91.116.13:4154 , piece_id=EVWXMFNELJCRDW5S3C5N6EWQE4QYHHTVMQ5FX5FVY2K6ZHPLZMZQ, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.051241009 (535h52m11.922324044s) [noise=true , address=77.90.31.13:4627 , piece_id=7ZMYWLFNSWTEDVUG2PZIIGFD2JA7ZT2VTDKFYJWBSWI36IB7EVLA, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.052314123 (535h52m11.923397158s) [noise=true , address=145.79.227.30:4761 , piece_id=B3X52A4XNTSLA5FWHWMTKT3XXUKMFRP64DSYIZ3CTOLZPCJWAFSA, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.054366752 (535h52m11.925449787s) [noise=true , address=145.79.227.9:4365 , piece_id=ELCBS5GXKBSYXP54CL7HMT54DHCCPCHTNTOGDSHAFCRKJI3Y6T2Q, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:13.761304484 (535h52m13.632387519s) [upload=469977]: Next returned nil segment Dec 27 23:21:13.761316758 (535h52m13.632399793s) [upload=469977]: Waiting for error group managing segments... Dec 27 23:21:14.044231686 (535h52m13.915314721s) [noise=true , address=145.79.227.32:4836 , piece_id=TQQ2WKMB3D7THEIQMQUZR7P6Y4G3JZHUDODCCF32YUQBSAJQPDAQ, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.044274999 (535h52m13.915358044s) [noise=true , address=145.79.227.16:4911 , piece_id=S757PSGJRTSR3ASTHD7BO6IDJSOXQJSL6Y2E73WN6I4P6Z5DS5BQ, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.048833579 (535h52m13.919916624s) [noise=true , address=145.79.227.15:4876 , piece_id=3NCIXSWK6KFUZLHMZDP7EL7YNIJKMGIKCF7LLKAU3QTQNMSZI3UA, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.076350486 (535h52m13.947433521s) [noise=true , address=145.79.227.9:4365 , piece_id=ELCBS5GXKBSYXP54CL7HMT54DHCCPCHTNTOGDSHAFCRKJI3Y6T2Q, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.104681974 (535h52m13.975765019s) [noise=true , address=145.79.227.4:4186 , piece_id=J5VBCZ2UIMKA2JVP7XJLC235WEX2CD6CMQNW6VSQD22G5URYGQFQ, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.110398509 (535h52m13.981481554s) [noise=true , address=145.79.227.19:4079 , piece_id=TJNOEVPGNYQ2QK3ZQS2HQTB4WGSWKVRPUGK5GYWPL7OQYPZOLNAA, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.221920787 (535h52m14.093003822s) [noise=true , address=77.90.31.12:4590 , piece_id=PK526LNTSML4FVUCJAOF45UNOGB3RCZZ3QJUXG7UKCP4NXTHOEGQ, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.227385763 (535h52m14.098468798s) [noise=true , address=77.90.31.13:4627 , piece_id=7ZMYWLFNSWTEDVUG2PZIIGFD2JA7ZT2VTDKFYJWBSWI36IB7EVLA, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.294875756 (535h52m14.165958801s) [noise=true , address=77.90.31.15:4697 , piece_id=34JFX6OGTQNV4NPQGVL5VS76Z5AFWXQAYFQSJFQFWCYAM5DU6WDA, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.339939444 (535h52m14.21102248s) [noise=true , address=45.91.116.7:4009 , piece_id=M5NUIJTA47GKJSLDTX2H4KN33NVI27CTDOA5MBQJP5JKQ7KQXJGA, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.617230693 (535h52m14.488313738s) [noise=true , address=77.90.31.11:4555 , piece_id=KG63ED3NRFLLNSR4AHGCTDH3JXFSWIJTS42LGB4X2PFG63L3TEEA, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:15.304314951 (535h52m15.175397996s) [noise=true , address=145.79.227.30:4761 , piece_id=B3X52A4XNTSLA5FWHWMTKT3XXUKMFRP64DSYIZ3CTOLZPCJWAFSA, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:15.349171756 (535h52m15.220254801s) [noise=true , address=45.91.116.11:4119 , piece_id=VWOWB6WBMHT7PXSM3DD2DEVFNUCSK2KDIXOX7LXIQW54EL6OW6SQ, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:15.349190050 (535h52m15.220273085s) [seg_pos={0 0}, upload=469977]: Segment reached optimal threshold of 13 pieces. Dec 27 23:21:15.349430779 (535h52m15.220513814s) [noise=true , address=145.79.227.33:4869 , piece_id=7QAVC4YREJMSCWN3OBHOH45XYBRTSALY3GLZGSPHQUIZVWBH7GVA, seg_pos={0 0}, upload=469977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 23:21:15.349436820 (535h52m15.220519855s) [noise=true , address=145.79.227.33:4869 , piece_id=7QAVC4YREJMSCWN3OBHOH45XYBRTSALY3GLZGSPHQUIZVWBH7GVA, seg_pos={0 0}, upload=469977]: Long tail cancellation due to optimal threshold Dec 27 23:21:15.349562590 (535h52m15.220645625s) [noise=true , address=45.91.116.13:4154 , piece_id=EVWXMFNELJCRDW5S3C5N6EWQE4QYHHTVMQ5FX5FVY2K6ZHPLZMZQ, seg_pos={0 0}, upload=469977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 23:21:15.349567980 (535h52m15.220651015s) [noise=true , address=45.91.116.13:4154 , piece_id=EVWXMFNELJCRDW5S3C5N6EWQE4QYHHTVMQ5FX5FVY2K6ZHPLZMZQ, seg_pos={0 0}, upload=469977]: Long tail cancellation due to optimal threshold Dec 27 23:21:15.349586997 (535h52m15.220670032s) [noise=true , address=45.91.116.14:4189 , piece_id=JH6Z2FA3WMVNIRUUXEDFAVY374PYT6YLZN3ZJLYJBD4JQFPXR7AA, seg_pos={0 0}, upload=469977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: read tcp 188.166.225.170:43448->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 23:21:15.349602426 (535h52m15.220685461s) [noise=true , address=45.91.116.14:4189 , piece_id=JH6Z2FA3WMVNIRUUXEDFAVY374PYT6YLZN3ZJLYJBD4JQFPXR7AA, seg_pos={0 0}, upload=469977]: Long tail cancellation due to optimal threshold Dec 27 23:21:15.349622845 (535h52m15.22070589s) [seg_pos={0 0}, upload=469977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:21:15.349699421 (535h52m15.220782456s) [upload=469977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:21:15.349705312 (535h52m15.220788347s) [upload=469977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:21:15.636205978 (535h52m15.507289013s) [upload=469977]: Done uploading segments... Dec 27 23:21:15.636224304 (535h52m15.507307339s) [upload=469977]: Upload finished. err: <nil> Duration: 7.400498078s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.X..X....X.......X.....X..........................XXX.....XX] duration: 30.00258331s trace: https://jaeger.dev.storj.io/trace/191f2761b4f31933 FAIL (2025-12-27 23:21:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 23:21:11.791799688 (535h52m11.662882733s) [upload=469977]: Starting upload Dec 27 23:21:11.791906573 (535h52m11.662989628s) [upload=469977]: Done starting upload Dec 27 23:21:11.791961737 (535h52m11.663044772s) [upload=469977]: Uploading segments... Dec 27 23:21:11.792283730 (535h52m11.663366785s) [upload=469977]: Got next segment. Inline: false Dec 27 23:21:11.792343194 (535h52m11.663426249s) [upload=469977]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:21:11.792349646 (535h52m11.663432701s) [upload=469977]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:21:12.038302212 (535h52m11.909385257s) [seg_pos={0 0}, upload=469977]: Begin upload segment... Dec 27 23:21:12.038347408 (535h52m11.909430453s) [seg_pos={0 0}, upload=469977]: Done begin upload segment. Dec 27 23:21:12.038370653 (535h52m11.909453688s) [noise=true , address=145.79.227.19:4079 , piece_id=TJNOEVPGNYQ2QK3ZQS2HQTB4WGSWKVRPUGK5GYWPL7OQYPZOLNAA, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.038382235 (535h52m11.90946527s) [noise=true , address=77.90.31.11:4555 , piece_id=KG63ED3NRFLLNSR4AHGCTDH3JXFSWIJTS42LGB4X2PFG63L3TEEA, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.038494097 (535h52m11.909577132s) [noise=true , address=77.90.31.15:4697 , piece_id=34JFX6OGTQNV4NPQGVL5VS76Z5AFWXQAYFQSJFQFWCYAM5DU6WDA, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.038775755 (535h52m11.9098588s) [noise=true , address=145.79.227.33:4869 , piece_id=7QAVC4YREJMSCWN3OBHOH45XYBRTSALY3GLZGSPHQUIZVWBH7GVA, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.039052531 (535h52m11.910135566s) [noise=true , address=45.91.116.14:4189 , piece_id=JH6Z2FA3WMVNIRUUXEDFAVY374PYT6YLZN3ZJLYJBD4JQFPXR7AA, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.039310683 (535h52m11.910393718s) [noise=true , address=77.90.31.12:4590 , piece_id=PK526LNTSML4FVUCJAOF45UNOGB3RCZZ3QJUXG7UKCP4NXTHOEGQ, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.039598931 (535h52m11.910681966s) [noise=true , address=45.91.116.7:4009 , piece_id=M5NUIJTA47GKJSLDTX2H4KN33NVI27CTDOA5MBQJP5JKQ7KQXJGA, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.039908381 (535h52m11.910991416s) [noise=true , address=45.91.116.11:4119 , piece_id=VWOWB6WBMHT7PXSM3DD2DEVFNUCSK2KDIXOX7LXIQW54EL6OW6SQ, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.042145621 (535h52m11.913228676s) [noise=true , address=145.79.227.4:4186 , piece_id=J5VBCZ2UIMKA2JVP7XJLC235WEX2CD6CMQNW6VSQD22G5URYGQFQ, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.043207333 (535h52m11.914290368s) [noise=true , address=145.79.227.32:4836 , piece_id=TQQ2WKMB3D7THEIQMQUZR7P6Y4G3JZHUDODCCF32YUQBSAJQPDAQ, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.044041381 (535h52m11.915124416s) [noise=true , address=145.79.227.15:4876 , piece_id=3NCIXSWK6KFUZLHMZDP7EL7YNIJKMGIKCF7LLKAU3QTQNMSZI3UA, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.047319984 (535h52m11.918403019s) [noise=true , address=145.79.227.16:4911 , piece_id=S757PSGJRTSR3ASTHD7BO6IDJSOXQJSL6Y2E73WN6I4P6Z5DS5BQ, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.049446333 (535h52m11.920529378s) [noise=true , address=45.91.116.13:4154 , piece_id=EVWXMFNELJCRDW5S3C5N6EWQE4QYHHTVMQ5FX5FVY2K6ZHPLZMZQ, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.051241009 (535h52m11.922324044s) [noise=true , address=77.90.31.13:4627 , piece_id=7ZMYWLFNSWTEDVUG2PZIIGFD2JA7ZT2VTDKFYJWBSWI36IB7EVLA, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.052314123 (535h52m11.923397158s) [noise=true , address=145.79.227.30:4761 , piece_id=B3X52A4XNTSLA5FWHWMTKT3XXUKMFRP64DSYIZ3CTOLZPCJWAFSA, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:12.054366752 (535h52m11.925449787s) [noise=true , address=145.79.227.9:4365 , piece_id=ELCBS5GXKBSYXP54CL7HMT54DHCCPCHTNTOGDSHAFCRKJI3Y6T2Q, seg_pos={0 0}, upload=469977]: Uploading piece... Dec 27 23:21:13.761304484 (535h52m13.632387519s) [upload=469977]: Next returned nil segment Dec 27 23:21:13.761316758 (535h52m13.632399793s) [upload=469977]: Waiting for error group managing segments... Dec 27 23:21:14.044231686 (535h52m13.915314721s) [noise=true , address=145.79.227.32:4836 , piece_id=TQQ2WKMB3D7THEIQMQUZR7P6Y4G3JZHUDODCCF32YUQBSAJQPDAQ, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.044274999 (535h52m13.915358044s) [noise=true , address=145.79.227.16:4911 , piece_id=S757PSGJRTSR3ASTHD7BO6IDJSOXQJSL6Y2E73WN6I4P6Z5DS5BQ, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.048833579 (535h52m13.919916624s) [noise=true , address=145.79.227.15:4876 , piece_id=3NCIXSWK6KFUZLHMZDP7EL7YNIJKMGIKCF7LLKAU3QTQNMSZI3UA, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.076350486 (535h52m13.947433521s) [noise=true , address=145.79.227.9:4365 , piece_id=ELCBS5GXKBSYXP54CL7HMT54DHCCPCHTNTOGDSHAFCRKJI3Y6T2Q, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.104681974 (535h52m13.975765019s) [noise=true , address=145.79.227.4:4186 , piece_id=J5VBCZ2UIMKA2JVP7XJLC235WEX2CD6CMQNW6VSQD22G5URYGQFQ, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.110398509 (535h52m13.981481554s) [noise=true , address=145.79.227.19:4079 , piece_id=TJNOEVPGNYQ2QK3ZQS2HQTB4WGSWKVRPUGK5GYWPL7OQYPZOLNAA, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.221920787 (535h52m14.093003822s) [noise=true , address=77.90.31.12:4590 , piece_id=PK526LNTSML4FVUCJAOF45UNOGB3RCZZ3QJUXG7UKCP4NXTHOEGQ, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.227385763 (535h52m14.098468798s) [noise=true , address=77.90.31.13:4627 , piece_id=7ZMYWLFNSWTEDVUG2PZIIGFD2JA7ZT2VTDKFYJWBSWI36IB7EVLA, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.294875756 (535h52m14.165958801s) [noise=true , address=77.90.31.15:4697 , piece_id=34JFX6OGTQNV4NPQGVL5VS76Z5AFWXQAYFQSJFQFWCYAM5DU6WDA, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.339939444 (535h52m14.21102248s) [noise=true , address=45.91.116.7:4009 , piece_id=M5NUIJTA47GKJSLDTX2H4KN33NVI27CTDOA5MBQJP5JKQ7KQXJGA, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:14.617230693 (535h52m14.488313738s) [noise=true , address=77.90.31.11:4555 , piece_id=KG63ED3NRFLLNSR4AHGCTDH3JXFSWIJTS42LGB4X2PFG63L3TEEA, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:15.304314951 (535h52m15.175397996s) [noise=true , address=145.79.227.30:4761 , piece_id=B3X52A4XNTSLA5FWHWMTKT3XXUKMFRP64DSYIZ3CTOLZPCJWAFSA, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:15.349171756 (535h52m15.220254801s) [noise=true , address=45.91.116.11:4119 , piece_id=VWOWB6WBMHT7PXSM3DD2DEVFNUCSK2KDIXOX7LXIQW54EL6OW6SQ, seg_pos={0 0}, upload=469977]: Done uploading piece. err: <nil> Dec 27 23:21:15.349190050 (535h52m15.220273085s) [seg_pos={0 0}, upload=469977]: Segment reached optimal threshold of 13 pieces. Dec 27 23:21:15.349430779 (535h52m15.220513814s) [noise=true , address=145.79.227.33:4869 , piece_id=7QAVC4YREJMSCWN3OBHOH45XYBRTSALY3GLZGSPHQUIZVWBH7GVA, seg_pos={0 0}, upload=469977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 23:21:15.349436820 (535h52m15.220519855s) [noise=true , address=145.79.227.33:4869 , piece_id=7QAVC4YREJMSCWN3OBHOH45XYBRTSALY3GLZGSPHQUIZVWBH7GVA, seg_pos={0 0}, upload=469977]: Long tail cancellation due to optimal threshold Dec 27 23:21:15.349562590 (535h52m15.220645625s) [noise=true , address=45.91.116.13:4154 , piece_id=EVWXMFNELJCRDW5S3C5N6EWQE4QYHHTVMQ5FX5FVY2K6ZHPLZMZQ, seg_pos={0 0}, upload=469977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 27 23:21:15.349567980 (535h52m15.220651015s) [noise=true , address=45.91.116.13:4154 , piece_id=EVWXMFNELJCRDW5S3C5N6EWQE4QYHHTVMQ5FX5FVY2K6ZHPLZMZQ, seg_pos={0 0}, upload=469977]: Long tail cancellation due to optimal threshold Dec 27 23:21:15.349586997 (535h52m15.220670032s) [noise=true , address=45.91.116.14:4189 , piece_id=JH6Z2FA3WMVNIRUUXEDFAVY374PYT6YLZN3ZJLYJBD4JQFPXR7AA, seg_pos={0 0}, upload=469977]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: read tcp 188.166.225.170:43448->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 23:21:15.349602426 (535h52m15.220685461s) [noise=true , address=45.91.116.14:4189 , piece_id=JH6Z2FA3WMVNIRUUXEDFAVY374PYT6YLZN3ZJLYJBD4JQFPXR7AA, seg_pos={0 0}, upload=469977]: Long tail cancellation due to optimal threshold Dec 27 23:21:15.349622845 (535h52m15.22070589s) [seg_pos={0 0}, upload=469977]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:21:15.349699421 (535h52m15.220782456s) [upload=469977]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:21:15.349705312 (535h52m15.220788347s) [upload=469977]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:21:15.636205978 (535h52m15.507289013s) [upload=469977]: Done uploading segments... Dec 27 23:21:15.636224304 (535h52m15.507307339s) [upload=469977]: Upload finished. err: <nil> Duration: 30.002591896s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 4.359353579s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:21:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.359373285s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X..X....X.......X.....X..........................XXX.....X] duration: 30.002551749s trace: https://jaeger.dev.storj.io/trace/7b395a65050a4bb1 FAIL (2025-12-27 23:21:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: 5XY5APCOALUIV4MOURCFTIOHTSAG2PWXRRCIYUEPPOEARW62LZTA) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: K2U2OSUO55OZSWASU7PH7F2IFIANY3DHT6QF3ZTN4EUG3V3J33QQ) context deadline exceeded; piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: MMA2I32GPHP3BPW3V3LF2X4O434Z6IEOLSZMTYJSCSDF7ROELR3Q) context deadline exceeded ======LOGS====== Dec 27 23:20:41.546619243 (535h51m41.417702278s) [upload=469971]: Got next segment. Inline: false Dec 27 23:20:41.546673195 (535h51m41.41775623s) [upload=469971]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:20:41.546679056 (535h51m41.417762091s) [upload=469971]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:20:41.790525561 (535h51m41.661608596s) [seg_pos={0 0}, upload=469971]: Begin upload segment... Dec 27 23:20:41.790584894 (535h51m41.661667929s) [seg_pos={0 0}, upload=469971]: Done begin upload segment. Dec 27 23:20:41.790612717 (535h51m41.661695752s) [noise=true , address=145.79.227.24:4581 , piece_id=3H44TP4X6YB4SKJYOUYZMAIT4L34J4454GN7NPE2VEANWFSR2XXQ, seg_pos={0 0}, upload=469971]: Uploading piece... Dec 27 23:20:41.790991418 (535h51m41.662074463s) [noise=true , address=45.91.116.9:4262 , piece_id=5XY5APCOALUIV4MOURCFTIOHTSAG2PWXRRCIYUEPPOEARW62LZTA, seg_pos={0 0}, upload=469971]: Uploading piece... Dec 27 23:20:41.791723082 (535h51m41.662806117s) [noise=true , address=145.79.227.19:4078 , piece_id=KXNHIWVS5M5X77FBXCFTZBQMYVENUVGSO2BGY7DMUGCK5PR4MOKA, seg_pos={0 0}, upload=469971]: Uploading piece... Dec 27 23:20:41.791899988 (535h51m41.662983023s) [noise=true , address=45.91.116.8:4048 , piece_id=K2U2OSUO55OZSWASU7PH7F2IFIANY3DHT6QF3ZTN4EUG3V3J33QQ, seg_pos={0 0}, upload=469971]: Uploading piece... Dec 27 23:20:41.792037881 (535h51m41.663120926s) [noise=true , address=45.91.116.12:4225 , piece_id=MYZPHJNEN7MNBU43KO4KEOUYDWB57L34WU4N7RINJ64BKRI2A42A, seg_pos={0 0}, upload=469971]: Uploading piece... Dec 27 23:20:41.792312173 (535h51m41.663395208s) [noise=true , address=45.91.116.6:4974 , piece_id=MMA2I32GPHP3BPW3V3LF2X4O434Z6IEOLSZMTYJSCSDF7ROELR3Q, seg_pos={0 0}, upload=469971]: Uploading piece... Dec 27 23:20:41.792488108 (535h51m41.663571153s) [noise=true , address=145.79.227.17:4971 , piece_id=OBVUHDNYMZDYEE4OY7SVDMDHUEM2OYRWVPUR33EQ6D5HKZROFGDQ, seg_pos={0 0}, upload=469971]: Uploading piece... Dec 27 23:20:41.792581435 (535h51m41.66366447s) [noise=true , address=145.79.227.9:4365 , piece_id=GJTZEIQOMI65EBAHMLKQUN45FBLCIYWKAURLFKXJMQEQPRQ77QSA, seg_pos={0 0}, upload=469971]: Uploading piece... Dec 27 23:20:41.792848214 (535h51m41.663931249s) [noise=true , address=77.90.31.11:4555 , piece_id=TE6AYOTIK3T3JCWQLSC4IEYUWWIJDWIHL5SHJRJW3QC5GHHLN2IA, seg_pos={0 0}, upload=469971]: Uploading piece... Dec 27 23:20:41.793015502 (535h51m41.664098547s) [noise=true , address=45.91.116.7:4011 , piece_id=XD62TK6HIJPB7274R5LAQ47FJS4XPXBPTLYP7XITGIVGNQBQ2KGQ, seg_pos={0 0}, upload=469971]: Uploading piece... Dec 27 23:20:41.793154687 (535h51m41.664237722s) [noise=true , address=145.79.227.52:4445 , piece_id=L6QKOOCKKM73MA6X6ONMFV5I6ZMZCCMTT4YS32RFDCEVUFUVBLIQ, seg_pos={0 0}, upload=469971]: Uploading piece... Dec 27 23:20:41.793459388 (535h51m41.664542433s) [noise=true , address=77.90.31.14:4662 , piece_id=XVEJLP2G5VSIE3HGUMRAJZZKL4GSHKI2YMKGNCKD7IR5F75UIACA, seg_pos={0 0}, upload=469971]: Uploading piece... Dec 27 23:20:41.793583173 (535h51m41.664666218s) [noise=true , address=77.90.31.17:4770 , piece_id=LZPDB6QLTPQUCHYNJTNCW2ZWJWSZMZU4ZM5DAKJ6OXXUSSNM3TNA, seg_pos={0 0}, upload=469971]: Uploading piece... Dec 27 23:20:41.793742457 (535h51m41.664825492s) [noise=true , address=145.79.227.51:4410 , piece_id=2IOSRSGKNJVH4Z2VHMTBO6WZZTCRY66PHSEFJ3NXI2OSRELU5CZQ, seg_pos={0 0}, upload=469971]: Uploading piece... Dec 27 23:20:41.794246706 (535h51m41.665329752s) [noise=true , address=145.79.227.46:4601 , piece_id=JQHHHH3KLT7OJ5NCPXJLVV64GJ6JYCXX3W42O3V4EQYKGBTLX3MQ, seg_pos={0 0}, upload=469971]: Uploading piece... Dec 27 23:20:41.794806783 (535h51m41.665889818s) [noise=true , address=45.91.116.14:4190 , piece_id=Y4ANQWHVK3QJJP3Q6UPTNZENWHAM3VDIHJL2T43ET4OOXOGJOVBA, seg_pos={0 0}, upload=469971]: Uploading piece... Dec 27 23:20:43.508562197 (535h51m43.379645242s) [upload=469971]: Next returned nil segment Dec 27 23:20:43.508574762 (535h51m43.379657797s) [upload=469971]: Waiting for error group managing segments... Dec 27 23:20:43.793156424 (535h51m43.664239469s) [noise=true , address=145.79.227.51:4410 , piece_id=2IOSRSGKNJVH4Z2VHMTBO6WZZTCRY66PHSEFJ3NXI2OSRELU5CZQ, seg_pos={0 0}, upload=469971]: Done uploading piece. err: <nil> Dec 27 23:20:43.795057694 (535h51m43.666140729s) [noise=true , address=145.79.227.19:4078 , piece_id=KXNHIWVS5M5X77FBXCFTZBQMYVENUVGSO2BGY7DMUGCK5PR4MOKA, seg_pos={0 0}, upload=469971]: Done uploading piece. err: <nil> Dec 27 23:20:43.801427153 (535h51m43.672510188s) [noise=true , address=145.79.227.9:4365 , piece_id=GJTZEIQOMI65EBAHMLKQUN45FBLCIYWKAURLFKXJMQEQPRQ77QSA, seg_pos={0 0}, upload=469971]: Done uploading piece. err: <nil> Dec 27 23:20:43.810909860 (535h51m43.681992895s) [noise=true , address=145.79.227.46:4601 , piece_id=JQHHHH3KLT7OJ5NCPXJLVV64GJ6JYCXX3W42O3V4EQYKGBTLX3MQ, seg_pos={0 0}, upload=469971]: Done uploading piece. err: <nil> Dec 27 23:20:43.816872985 (535h51m43.68795602s) [noise=true , address=145.79.227.52:4445 , piece_id=L6QKOOCKKM73MA6X6ONMFV5I6ZMZCCMTT4YS32RFDCEVUFUVBLIQ, seg_pos={0 0}, upload=469971]: Done uploading piece. err: <nil> Dec 27 23:20:44.017404250 (535h51m43.888487285s) [noise=true , address=77.90.31.17:4770 , piece_id=LZPDB6QLTPQUCHYNJTNCW2ZWJWSZMZU4ZM5DAKJ6OXXUSSNM3TNA, seg_pos={0 0}, upload=469971]: Done uploading piece. err: <nil> Dec 27 23:20:44.027794325 (535h51m43.89887736s) [noise=true , address=77.90.31.14:4662 , piece_id=XVEJLP2G5VSIE3HGUMRAJZZKL4GSHKI2YMKGNCKD7IR5F75UIACA, seg_pos={0 0}, upload=469971]: Done uploading piece. err: <nil> Dec 27 23:20:44.044292060 (535h51m43.915375095s) [noise=true , address=45.91.116.7:4011 , piece_id=XD62TK6HIJPB7274R5LAQ47FJS4XPXBPTLYP7XITGIVGNQBQ2KGQ, seg_pos={0 0}, upload=469971]: Done uploading piece. err: <nil> Dec 27 23:20:44.051440121 (535h51m43.922523156s) [noise=true , address=45.91.116.8:4048 , piece_id=K2U2OSUO55OZSWASU7PH7F2IFIANY3DHT6QF3ZTN4EUG3V3J33QQ, seg_pos={0 0}, upload=469971]: Done uploading piece. err: <nil> Dec 27 23:20:44.060799432 (535h51m43.931882467s) [noise=true , address=45.91.116.6:4974 , piece_id=MMA2I32GPHP3BPW3V3LF2X4O434Z6IEOLSZMTYJSCSDF7ROELR3Q, seg_pos={0 0}, upload=469971]: Done uploading piece. err: <nil> Dec 27 23:20:44.078792745 (535h51m43.94987578s) [noise=true , address=77.90.31.11:4555 , piece_id=TE6AYOTIK3T3JCWQLSC4IEYUWWIJDWIHL5SHJRJW3QC5GHHLN2IA, seg_pos={0 0}, upload=469971]: Done uploading piece. err: <nil> Dec 27 23:20:44.105597376 (535h51m43.976680421s) [noise=true , address=45.91.116.14:4190 , piece_id=Y4ANQWHVK3QJJP3Q6UPTNZENWHAM3VDIHJL2T43ET4OOXOGJOVBA, seg_pos={0 0}, upload=469971]: Done uploading piece. err: <nil> Dec 27 23:20:44.460008504 (535h51m44.331091539s) [noise=true , address=45.91.116.9:4262 , piece_id=5XY5APCOALUIV4MOURCFTIOHTSAG2PWXRRCIYUEPPOEARW62LZTA, seg_pos={0 0}, upload=469971]: Done uploading piece. err: <nil> Dec 27 23:20:44.460027932 (535h51m44.331110967s) [seg_pos={0 0}, upload=469971]: Segment reached optimal threshold of 13 pieces. Dec 27 23:20:44.460199799 (535h51m44.331282834s) [noise=true , address=145.79.227.17:4971 , piece_id=OBVUHDNYMZDYEE4OY7SVDMDHUEM2OYRWVPUR33EQ6D5HKZROFGDQ, seg_pos={0 0}, upload=469971]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:45590->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:45590->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45590->145.79.227.17:4971: use of closed network connection Dec 27 23:20:44.460205169 (535h51m44.331288204s) [noise=true , address=145.79.227.17:4971 , piece_id=OBVUHDNYMZDYEE4OY7SVDMDHUEM2OYRWVPUR33EQ6D5HKZROFGDQ, seg_pos={0 0}, upload=469971]: Long tail cancellation due to optimal threshold Dec 27 23:20:44.460273119 (535h51m44.331356154s) [noise=true , address=45.91.116.12:4225 , piece_id=MYZPHJNEN7MNBU43KO4KEOUYDWB57L34WU4N7RINJ64BKRI2A42A, seg_pos={0 0}, upload=469971]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 27 23:20:44.460281895 (535h51m44.33136493s) [noise=true , address=45.91.116.12:4225 , piece_id=MYZPHJNEN7MNBU43KO4KEOUYDWB57L34WU4N7RINJ64BKRI2A42A, seg_pos={0 0}, upload=469971]: Long tail cancellation due to optimal threshold Dec 27 23:20:44.460424477 (535h51m44.331507512s) [noise=true , address=145.79.227.24:4581 , piece_id=3H44TP4X6YB4SKJYOUYZMAIT4L34J4454GN7NPE2VEANWFSR2XXQ, seg_pos={0 0}, upload=469971]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123ib1A8nmVmR272odjhVsUgADYbaMRMiPahSLoZ5eamaV6Qbb1): protocol: write tcp 188.166.225.170:58780->145.79.227.24:4581: use of closed network connection; write tcp 188.166.225.170:58780->145.79.227.24:4581: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58780->145.79.227.24:4581: use of closed network connection Dec 27 23:20:44.460430168 (535h51m44.331513203s) [noise=true , address=145.79.227.24:4581 , piece_id=3H44TP4X6YB4SKJYOUYZMAIT4L34J4454GN7NPE2VEANWFSR2XXQ, seg_pos={0 0}, upload=469971]: Long tail cancellation due to optimal threshold Dec 27 23:20:44.460445747 (535h51m44.331528782s) [seg_pos={0 0}, upload=469971]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:20:44.460505231 (535h51m44.331588266s) [upload=469971]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:20:44.460509549 (535h51m44.331592584s) [upload=469971]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:20:44.757339343 (535h51m44.628422378s) [upload=469971]: Done uploading segments... Dec 27 23:20:44.757357017 (535h51m44.628440063s) [upload=469971]: Upload finished. err: <nil> Duration: 30.002576227s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 4.139331217s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:20:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.139360853s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X..X....X.......X.....X..........................XXX.....] duration: 28.756097615s trace: https://jaeger.dev.storj.io/trace/27ab06645e4f2929 FAIL (2025-12-27 23:20:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: JXQRL54TGDNYN5JNOVEA3ICDGK42OHG2NTC3PT7WZOBMSTNO6GYA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: UZ2Y4FIO5ROHXYNAH5NLMIMZ3QVW5PI5LU7LQFQKUF4BCX3INJCQ) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: VIBII6J5AW552SRX4CCFP3NSVU3T7L6LIDJ3TINB4HFDHBU4EEKQ) context deadline exceeded ======LOGS====== :16:26.396195504 (535h47m26.267278539s) [upload=469908]: Starting upload Dec 27 23:16:26.396255629 (535h47m26.267338664s) [upload=469908]: Done starting upload Dec 27 23:16:26.396314431 (535h47m26.267397466s) [upload=469908]: Uploading segments... Dec 27 23:16:26.396548426 (535h47m26.267631471s) [upload=469908]: Got next segment. Inline: false Dec 27 23:16:26.396600516 (535h47m26.267683551s) [upload=469908]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 23:16:26.396606036 (535h47m26.267689081s) [upload=469908]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 23:16:26.644698036 (535h47m26.515781071s) [seg_pos={0 0}, upload=469908]: Begin upload segment... Dec 27 23:16:26.644747871 (535h47m26.515830916s) [seg_pos={0 0}, upload=469908]: Done begin upload segment. Dec 27 23:16:26.644773610 (535h47m26.515856645s) [noise=true , address=145.79.227.19:4078 , piece_id=EVTDCQG77KJWHF2UA5BXCYZ5SLA6ZORL3C7KHNHZP3HDSFQA5ZVQ, seg_pos={0 0}, upload=469908]: Uploading piece... Dec 27 23:16:26.644933635 (535h47m26.51601669s) [noise=true , address=145.79.227.46:4603 , piece_id=KXFEQ5P4QDBEC6F6RSVGEZQEXX74DU3X7AIPCRHZICOZIXYA6MRQ, seg_pos={0 0}, upload=469908]: Uploading piece... Dec 27 23:16:26.645614631 (535h47m26.516697676s) [noise=true , address=77.90.31.10:4518 , piece_id=JXQRL54TGDNYN5JNOVEA3ICDGK42OHG2NTC3PT7WZOBMSTNO6GYA, seg_pos={0 0}, upload=469908]: Uploading piece... Dec 27 23:16:26.646035643 (535h47m26.517118688s) [noise=true , address=45.91.116.13:4156 , piece_id=UZ2Y4FIO5ROHXYNAH5NLMIMZ3QVW5PI5LU7LQFQKUF4BCX3INJCQ, seg_pos={0 0}, upload=469908]: Uploading piece... Dec 27 23:16:26.646436126 (535h47m26.517519161s) [noise=true , address=77.90.31.12:4590 , piece_id=VIBII6J5AW552SRX4CCFP3NSVU3T7L6LIDJ3TINB4HFDHBU4EEKQ, seg_pos={0 0}, upload=469908]: Uploading piece... Dec 27 23:16:26.646510698 (535h47m26.517593743s) [noise=true , address=77.90.31.18:4805 , piece_id=QTXHGAEIVLZQRGWFE6R6OGKHD2AP3PHD4COR3RILYU7FVPMSULEA, seg_pos={0 0}, upload=469908]: Uploading piece... Dec 27 23:16:26.646837560 (535h47m26.517920605s) [noise=true , address=45.91.116.6:4975 , piece_id=UIF5UO5XYM5O2U6FWWPQLUE75ANA5H7JYJR5PVX2OKGB7X7V6I3Q, seg_pos={0 0}, upload=469908]: Uploading piece... Dec 27 23:16:26.647116421 (535h47m26.518199456s) [noise=true , address=77.90.31.14:4662 , piece_id=JLSEBZ6FFISNO57MZPMXKVXFLPBFTHTTWVXBUHAFCW2SIG2KE6HQ, seg_pos={0 0}, upload=469908]: Uploading piece... Dec 27 23:16:26.647373220 (535h47m26.518456265s) [noise=true , address=145.79.227.39:4015 , piece_id=T45N3FMM3LCNNPTPG5S5AMS6EBFT2EF4VC6JWYJK2DFMHPEVRZEA, seg_pos={0 0}, upload=469908]: Uploading piece... Dec 27 23:16:26.647382598 (535h47m26.518465633s) [noise=true , address=145.79.227.13:4803 , piece_id=PHMGBMTSF6TNLRIHZIJ44663JJZM65XCTHQ6RV6Z7PVGN4L2RBHA, seg_pos={0 0}, upload=469908]: Uploading piece... Dec 27 23:16:26.647668242 (535h47m26.518751277s) [noise=true , address=145.79.227.11:4731 , piece_id=VMFK2DWNMBTIR6QXUC3GOTQA5WXCC7J5MSOS3C4GKCYYVQB7GNZA, seg_pos={0 0}, upload=469908]: Uploading piece... Dec 27 23:16:26.647929520 (535h47m26.519012555s) [noise=true , address=145.79.227.50:4375 , piece_id=AI3MVA7LBP7CRRAYGUXISFOWOAF3L2T7VKEUKZN4CJR3FDAIVWMQ, seg_pos={0 0}, upload=469908]: Uploading piece... Dec 27 23:16:26.648045520 (535h47m26.519128555s) [noise=true , address=45.91.116.8:4047 , piece_id=MZD7GRCGLSV62VLEIVSXUT2BE7MSICHKWRXWYLUHN32JM2ZNTYSQ, seg_pos={0 0}, upload=469908]: Uploading piece... Dec 27 23:16:26.648194313 (535h47m26.519277358s) [noise=true , address=145.79.227.45:4231 , piece_id=TPJ5OC5HTOPYNOH3PS5TU277LWEPIMIUVD7GXTVVNCPKJEQZWMFA, seg_pos={0 0}, upload=469908]: Uploading piece... Dec 27 23:16:26.648798394 (535h47m26.519881439s) [noise=true , address=145.79.227.4:4185 , piece_id=JCJ6G7SOOLRK5BDAND3HS2RRXE45T4XW7EIRTAJQREF6SMRIJJEQ, seg_pos={0 0}, upload=469908]: Uploading piece... Dec 27 23:16:26.649505811 (535h47m26.520588856s) [noise=true , address=45.91.116.5:4297 , piece_id=R32AAQLAGVGAR4UQF42CAWNB6YNERM25HWTTJX2L3IXXEWG7IF5A, seg_pos={0 0}, upload=469908]: Uploading piece... Dec 27 23:16:28.278918362 (535h47m28.150001397s) [upload=469908]: Next returned nil segment Dec 27 23:16:28.278932239 (535h47m28.150015284s) [upload=469908]: Waiting for error group managing segments... Dec 27 23:16:28.642999004 (535h47m28.514082039s) [noise=true , address=145.79.227.45:4231 , piece_id=TPJ5OC5HTOPYNOH3PS5TU277LWEPIMIUVD7GXTVVNCPKJEQZWMFA, seg_pos={0 0}, upload=469908]: Done uploading piece. err: <nil> Dec 27 23:16:28.644062780 (535h47m28.515145815s) [noise=true , address=145.79.227.50:4375 , piece_id=AI3MVA7LBP7CRRAYGUXISFOWOAF3L2T7VKEUKZN4CJR3FDAIVWMQ, seg_pos={0 0}, upload=469908]: Done uploading piece. err: <nil> Dec 27 23:16:28.651471728 (535h47m28.522554763s) [noise=true , address=145.79.227.39:4015 , piece_id=T45N3FMM3LCNNPTPG5S5AMS6EBFT2EF4VC6JWYJK2DFMHPEVRZEA, seg_pos={0 0}, upload=469908]: Done uploading piece. err: <nil> Dec 27 23:16:28.664132917 (535h47m28.535215962s) [noise=true , address=145.79.227.46:4603 , piece_id=KXFEQ5P4QDBEC6F6RSVGEZQEXX74DU3X7AIPCRHZICOZIXYA6MRQ, seg_pos={0 0}, upload=469908]: Done uploading piece. err: <nil> Dec 27 23:16:28.664179375 (535h47m28.53526241s) [noise=true , address=145.79.227.11:4731 , piece_id=VMFK2DWNMBTIR6QXUC3GOTQA5WXCC7J5MSOS3C4GKCYYVQB7GNZA, seg_pos={0 0}, upload=469908]: Done uploading piece. err: <nil> Dec 27 23:16:28.667433461 (535h47m28.538516496s) [noise=true , address=145.79.227.4:4185 , piece_id=JCJ6G7SOOLRK5BDAND3HS2RRXE45T4XW7EIRTAJQREF6SMRIJJEQ, seg_pos={0 0}, upload=469908]: Done uploading piece. err: <nil> Dec 27 23:16:28.768937864 (535h47m28.640020899s) [noise=true , address=145.79.227.13:4803 , piece_id=PHMGBMTSF6TNLRIHZIJ44663JJZM65XCTHQ6RV6Z7PVGN4L2RBHA, seg_pos={0 0}, upload=469908]: Done uploading piece. err: <nil> Dec 27 23:16:28.836895075 (535h47m28.70797811s) [noise=true , address=77.90.31.14:4662 , piece_id=JLSEBZ6FFISNO57MZPMXKVXFLPBFTHTTWVXBUHAFCW2SIG2KE6HQ, seg_pos={0 0}, upload=469908]: Done uploading piece. err: <nil> Dec 27 23:16:28.841950953 (535h47m28.713033998s) [noise=true , address=77.90.31.10:4518 , piece_id=JXQRL54TGDNYN5JNOVEA3ICDGK42OHG2NTC3PT7WZOBMSTNO6GYA, seg_pos={0 0}, upload=469908]: Done uploading piece. err: <nil> Dec 27 23:16:28.890921401 (535h47m28.762004446s) [noise=true , address=45.91.116.8:4047 , piece_id=MZD7GRCGLSV62VLEIVSXUT2BE7MSICHKWRXWYLUHN32JM2ZNTYSQ, seg_pos={0 0}, upload=469908]: Done uploading piece. err: <nil> Dec 27 23:16:28.925220693 (535h47m28.796303728s) [noise=true , address=77.90.31.12:4590 , piece_id=VIBII6J5AW552SRX4CCFP3NSVU3T7L6LIDJ3TINB4HFDHBU4EEKQ, seg_pos={0 0}, upload=469908]: Done uploading piece. err: <nil> Dec 27 23:16:28.945723494 (535h47m28.816806529s) [noise=true , address=45.91.116.5:4297 , piece_id=R32AAQLAGVGAR4UQF42CAWNB6YNERM25HWTTJX2L3IXXEWG7IF5A, seg_pos={0 0}, upload=469908]: Done uploading piece. err: <nil> Dec 27 23:16:28.973042194 (535h47m28.844125229s) [noise=true , address=45.91.116.13:4156 , piece_id=UZ2Y4FIO5ROHXYNAH5NLMIMZ3QVW5PI5LU7LQFQKUF4BCX3INJCQ, seg_pos={0 0}, upload=469908]: Done uploading piece. err: <nil> Dec 27 23:16:28.973064455 (535h47m28.84414749s) [seg_pos={0 0}, upload=469908]: Segment reached optimal threshold of 13 pieces. Dec 27 23:16:28.973327086 (535h47m28.844410141s) [noise=true , address=145.79.227.19:4078 , piece_id=EVTDCQG77KJWHF2UA5BXCYZ5SLA6ZORL3C7KHNHZP3HDSFQA5ZVQ, seg_pos={0 0}, upload=469908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:34962->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:34962->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34962->145.79.227.19:4078: use of closed network connection Dec 27 23:16:28.973337165 (535h47m28.84442021s) [noise=true , address=145.79.227.19:4078 , piece_id=EVTDCQG77KJWHF2UA5BXCYZ5SLA6ZORL3C7KHNHZP3HDSFQA5ZVQ, seg_pos={0 0}, upload=469908]: Long tail cancellation due to optimal threshold Dec 27 23:16:28.973573796 (535h47m28.844656831s) [noise=true , address=77.90.31.18:4805 , piece_id=QTXHGAEIVLZQRGWFE6R6OGKHD2AP3PHD4COR3RILYU7FVPMSULEA, seg_pos={0 0}, upload=469908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF Dec 27 23:16:28.973582732 (535h47m28.844665767s) [noise=true , address=77.90.31.18:4805 , piece_id=QTXHGAEIVLZQRGWFE6R6OGKHD2AP3PHD4COR3RILYU7FVPMSULEA, seg_pos={0 0}, upload=469908]: Long tail cancellation due to optimal threshold Dec 27 23:16:28.973791229 (535h47m28.844874265s) [noise=true , address=45.91.116.6:4975 , piece_id=UIF5UO5XYM5O2U6FWWPQLUE75ANA5H7JYJR5PVX2OKGB7X7V6I3Q, seg_pos={0 0}, upload=469908]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 27 23:16:28.973858117 (535h47m28.844941152s) [noise=true , address=45.91.116.6:4975 , piece_id=UIF5UO5XYM5O2U6FWWPQLUE75ANA5H7JYJR5PVX2OKGB7X7V6I3Q, seg_pos={0 0}, upload=469908]: Long tail cancellation due to optimal threshold Dec 27 23:16:28.973881051 (535h47m28.844964086s) [seg_pos={0 0}, upload=469908]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 23:16:28.973981773 (535h47m28.845064808s) [upload=469908]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 23:16:28.973986682 (535h47m28.845069727s) [upload=469908]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 23:16:29.263245180 (535h47m29.134328285s) [upload=469908]: Done uploading segments... Dec 27 23:16:29.263270699 (535h47m29.134353734s) [upload=469908]: Upload finished. err: <nil> Duration: 28.756126219s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.603252722s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:19:51 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.603293099s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.613809184s trace: https://jaeger.dev.storj.io/trace/4bb85e67f0d4ec1d FAIL (2025-12-27 23:19:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.613834351s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)