Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [XX.XXX..X..X.........X.XX..X....X.XX.........X.XXX.XX.....X.] duration:
5.985235816s trace:
https://jaeger.dev.storj.io/trace/146989eab2ea58c7
FAIL (2025-12-28 06:50:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 6F4QPKTMU5DJDRAGBV6XS3JXSA6T2Y5WV2KYGS36SSWPFAR37XYA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: BL2KB2IMYCXC6YV64W4NUFTXMY3ZHQ54WRHQENSK5LVTCMAUVMJQ) context deadline exceeded
======LOGS======
load
Dec 28 06:49:59.297128816 (543h20m59.168211871s) [upload=476491]: Done starting upload
Dec 28 06:49:59.297215831 (543h20m59.168298886s) [upload=476491]: Uploading segments...
Dec 28 06:49:59.297449166 (543h20m59.168532211s) [upload=476491]: Got next segment. Inline: false
Dec 28 06:49:59.297510743 (543h20m59.168593798s) [upload=476491]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:49:59.297516845 (543h20m59.16859988s) [upload=476491]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:49:59.540204673 (543h20m59.411287719s) [seg_pos={0 0}, upload=476491]: Begin upload segment...
Dec 28 06:49:59.540254087 (543h20m59.411337122s) [seg_pos={0 0}, upload=476491]: Done begin upload segment.
Dec 28 06:49:59.540280478 (543h20m59.411363513s) [noise=true , address=45.91.116.11:4119 , piece_id=NGOLUFRHI3LNYUVX7W2NMFFZQNUGT7NTBZM5STOEDIKIX2KNB7OQ, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.540328299 (543h20m59.411411334s) [noise=true , address=145.79.227.14:4839 , piece_id=HLWMDQCJYBRWGVQNJHR6X7JEMI5FXZCELVHJK2OX2XZDLKUNONPQ, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.540988667 (543h20m59.412071712s) [noise=true , address=145.79.227.19:4078 , piece_id=I63FUO5ACRGWQGZRTSA7ELTWQT5ED5D3PEB2F7II5BGUKDY7GT6Q, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.541438833 (543h20m59.412521878s) [noise=true , address=145.79.227.38:4979 , piece_id=VEGJDYAE2ZNAQE2REEMMLRGTDLCSNLPJS7OYHUKJZBR5Q5IR5V4Q, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.541933876 (543h20m59.413016912s) [noise=true , address=145.79.227.46:4601 , piece_id=QJUNANFVIWS3RXHNSHQXV3QJLMLMVWSUJEXVRXEQ3IN4RF2BWGCA, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.542494925 (543h20m59.41357796s) [noise=true , address=77.90.31.10:4518 , piece_id=BL2KB2IMYCXC6YV64W4NUFTXMY3ZHQ54WRHQENSK5LVTCMAUVMJQ, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.542977033 (543h20m59.414060068s) [noise=true , address=145.79.227.13:4803 , piece_id=Y6J7IJAPDQYHALFN5O5ZQ67JURPO4EAUEUOXCUVI6E76TMFFZDLQ, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.542997411 (543h20m59.414080446s) [noise=true , address=45.91.116.13:4156 , piece_id=PGUEGBXASOPBNIODQ2G7SFBIYS25BH7EDRYPVPQ7VW4P4CFDI3TA, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.543381653 (543h20m59.414464688s) [noise=true , address=145.79.227.32:4835 , piece_id=TG7JI3BCO2Y2F2LHQAM6VEJYIKIENJ4PI6IYXA25K6K26XPUQ2KQ, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.543727081 (543h20m59.414810116s) [noise=true , address=145.79.227.6:4258 , piece_id=A3FIIPNCDDAM3O65WD5SFCVFBVLOFQHCH2U2ALP4I7KNPCEVFLLA, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.544084831 (543h20m59.415167877s) [noise=true , address=77.90.31.11:4554 , piece_id=U7YOB7RENTK43OCCNICZ7HZZFTSUTNXX5MLL2A3U6QNLXJXQEU5Q, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.544482139 (543h20m59.415565174s) [noise=true , address=45.91.116.12:4225 , piece_id=6F4QPKTMU5DJDRAGBV6XS3JXSA6T2Y5WV2KYGS36SSWPFAR37XYA, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.544826845 (543h20m59.41590988s) [noise=true , address=145.79.227.45:4230 , piece_id=36RAG2LPPCMAHMA7CGJPBN4JYOYIYIIGWJZCBKDGXW3JOTXL2RVQ, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.546119346 (543h20m59.417202391s) [noise=true , address=45.91.116.14:4189 , piece_id=KIPPHV5VAEGQ6PDIRQOGRP2OUCGMBTMDB4FU5CMY53DIQ34IOCTA, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.551669674 (543h20m59.42275272s) [noise=true , address=145.79.227.16:4909 , piece_id=OTDAYRDRJ3KND3GM7I3WMI3GLNSARWQZUFU2UZB5VSSQED6QUQGA, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.552830656 (543h20m59.423913691s) [noise=true , address=77.90.31.18:4808 , piece_id=532LOSBPEQNSBIMRW7FZM6DX5SBAILF3GDRNJGO7UIPBA5TG42FA, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:50:01.258365217 (543h21m1.129448292s) [upload=476491]: Next returned nil segment
Dec 28 06:50:01.258398810 (543h21m1.129481855s) [upload=476491]: Waiting for error group managing segments...
Dec 28 06:50:01.543372188 (543h21m1.414455233s) [noise=true , address=145.79.227.19:4078 , piece_id=I63FUO5ACRGWQGZRTSA7ELTWQT5ED5D3PEB2F7II5BGUKDY7GT6Q, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.565558916 (543h21m1.436641971s) [noise=true , address=145.79.227.13:4803 , piece_id=Y6J7IJAPDQYHALFN5O5ZQ67JURPO4EAUEUOXCUVI6E76TMFFZDLQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.576701373 (543h21m1.447784438s) [noise=true , address=145.79.227.6:4258 , piece_id=A3FIIPNCDDAM3O65WD5SFCVFBVLOFQHCH2U2ALP4I7KNPCEVFLLA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.579147681 (543h21m1.450230726s) [noise=true , address=145.79.227.38:4979 , piece_id=VEGJDYAE2ZNAQE2REEMMLRGTDLCSNLPJS7OYHUKJZBR5Q5IR5V4Q, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.585783266 (543h21m1.456866311s) [noise=true , address=145.79.227.32:4835 , piece_id=TG7JI3BCO2Y2F2LHQAM6VEJYIKIENJ4PI6IYXA25K6K26XPUQ2KQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.587092168 (543h21m1.458175213s) [noise=true , address=145.79.227.14:4839 , piece_id=HLWMDQCJYBRWGVQNJHR6X7JEMI5FXZCELVHJK2OX2XZDLKUNONPQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.588867348 (543h21m1.459950403s) [noise=true , address=145.79.227.46:4601 , piece_id=QJUNANFVIWS3RXHNSHQXV3QJLMLMVWSUJEXVRXEQ3IN4RF2BWGCA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.591090392 (543h21m1.462173437s) [noise=true , address=145.79.227.16:4909 , piece_id=OTDAYRDRJ3KND3GM7I3WMI3GLNSARWQZUFU2UZB5VSSQED6QUQGA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.591242731 (543h21m1.462325776s) [noise=true , address=145.79.227.45:4230 , piece_id=36RAG2LPPCMAHMA7CGJPBN4JYOYIYIIGWJZCBKDGXW3JOTXL2RVQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.718671558 (543h21m1.589754603s) [noise=true , address=77.90.31.10:4518 , piece_id=BL2KB2IMYCXC6YV64W4NUFTXMY3ZHQ54WRHQENSK5LVTCMAUVMJQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.786458726 (543h21m1.657541771s) [noise=true , address=45.91.116.12:4225 , piece_id=6F4QPKTMU5DJDRAGBV6XS3JXSA6T2Y5WV2KYGS36SSWPFAR37XYA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.954373975 (543h21m1.82545706s) [noise=true , address=45.91.116.11:4119 , piece_id=NGOLUFRHI3LNYUVX7W2NMFFZQNUGT7NTBZM5STOEDIKIX2KNB7OQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:02.201054377 (543h21m2.072137432s) [noise=true , address=77.90.31.11:4554 , piece_id=U7YOB7RENTK43OCCNICZ7HZZFTSUTNXX5MLL2A3U6QNLXJXQEU5Q, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:02.201346554 (543h21m2.072429609s) [seg_pos={0 0}, upload=476491]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:50:02.201691010 (543h21m2.072774055s) [noise=true , address=77.90.31.18:4808 , piece_id=532LOSBPEQNSBIMRW7FZM6DX5SBAILF3GDRNJGO7UIPBA5TG42FA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:44396->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:44396->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44396->77.90.31.18:4808: use of closed network connection
Dec 28 06:50:02.201702451 (543h21m2.072785497s) [noise=true , address=77.90.31.18:4808 , piece_id=532LOSBPEQNSBIMRW7FZM6DX5SBAILF3GDRNJGO7UIPBA5TG42FA, seg_pos={0 0}, upload=476491]: Long tail cancellation due to optimal threshold
Dec 28 06:50:02.202552920 (543h21m2.073635966s) [noise=true , address=45.91.116.13:4156 , piece_id=PGUEGBXASOPBNIODQ2G7SFBIYS25BH7EDRYPVPQ7VW4P4CFDI3TA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 28 06:50:02.202565415 (543h21m2.07364846s) [noise=true , address=45.91.116.13:4156 , piece_id=PGUEGBXASOPBNIODQ2G7SFBIYS25BH7EDRYPVPQ7VW4P4CFDI3TA, seg_pos={0 0}, upload=476491]: Long tail cancellation due to optimal threshold
Dec 28 06:50:02.202783540 (543h21m2.073866595s) [noise=true , address=45.91.116.14:4189 , piece_id=KIPPHV5VAEGQ6PDIRQOGRP2OUCGMBTMDB4FU5CMY53DIQ34IOCTA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: read tcp 188.166.225.170:34022->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 06:50:02.202796154 (543h21m2.073879199s) [noise=true , address=45.91.116.14:4189 , piece_id=KIPPHV5VAEGQ6PDIRQOGRP2OUCGMBTMDB4FU5CMY53DIQ34IOCTA, seg_pos={0 0}, upload=476491]: Long tail cancellation due to optimal threshold
Dec 28 06:50:02.202823887 (543h21m2.073906932s) [seg_pos={0 0}, upload=476491]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:50:02.202945759 (543h21m2.074028804s) [upload=476491]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:50:02.202954987 (543h21m2.074038032s) [upload=476491]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:50:02.492076845 (543h21m2.36315989s) [upload=476491]: Done uploading segments...
Dec 28 06:50:02.492103696 (543h21m2.363186751s) [upload=476491]: Upload finished. err: <nil>
Duration: 5.985254661s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [..XX.X...XX.....X................X.X.....X.X.........X.X...X] duration:
5.000814338s trace:
https://jaeger.dev.storj.io/trace/662c07037cb3ac40
FAIL (2025-12-28 06:50:33 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000918948s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.XX.XXX..X..X.........X.XX..X....X.XX.........X.XXX.XX.....X] duration:
30.003951326s trace:
https://jaeger.dev.storj.io/trace/146989eab2ea58c7
FAIL (2025-12-28 06:50:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: 6F4QPKTMU5DJDRAGBV6XS3JXSA6T2Y5WV2KYGS36SSWPFAR37XYA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: BL2KB2IMYCXC6YV64W4NUFTXMY3ZHQ54WRHQENSK5LVTCMAUVMJQ) context deadline exceeded
======LOGS======
load
Dec 28 06:49:59.297128816 (543h20m59.168211871s) [upload=476491]: Done starting upload
Dec 28 06:49:59.297215831 (543h20m59.168298886s) [upload=476491]: Uploading segments...
Dec 28 06:49:59.297449166 (543h20m59.168532211s) [upload=476491]: Got next segment. Inline: false
Dec 28 06:49:59.297510743 (543h20m59.168593798s) [upload=476491]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:49:59.297516845 (543h20m59.16859988s) [upload=476491]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:49:59.540204673 (543h20m59.411287719s) [seg_pos={0 0}, upload=476491]: Begin upload segment...
Dec 28 06:49:59.540254087 (543h20m59.411337122s) [seg_pos={0 0}, upload=476491]: Done begin upload segment.
Dec 28 06:49:59.540280478 (543h20m59.411363513s) [noise=true , address=45.91.116.11:4119 , piece_id=NGOLUFRHI3LNYUVX7W2NMFFZQNUGT7NTBZM5STOEDIKIX2KNB7OQ, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.540328299 (543h20m59.411411334s) [noise=true , address=145.79.227.14:4839 , piece_id=HLWMDQCJYBRWGVQNJHR6X7JEMI5FXZCELVHJK2OX2XZDLKUNONPQ, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.540988667 (543h20m59.412071712s) [noise=true , address=145.79.227.19:4078 , piece_id=I63FUO5ACRGWQGZRTSA7ELTWQT5ED5D3PEB2F7II5BGUKDY7GT6Q, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.541438833 (543h20m59.412521878s) [noise=true , address=145.79.227.38:4979 , piece_id=VEGJDYAE2ZNAQE2REEMMLRGTDLCSNLPJS7OYHUKJZBR5Q5IR5V4Q, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.541933876 (543h20m59.413016912s) [noise=true , address=145.79.227.46:4601 , piece_id=QJUNANFVIWS3RXHNSHQXV3QJLMLMVWSUJEXVRXEQ3IN4RF2BWGCA, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.542494925 (543h20m59.41357796s) [noise=true , address=77.90.31.10:4518 , piece_id=BL2KB2IMYCXC6YV64W4NUFTXMY3ZHQ54WRHQENSK5LVTCMAUVMJQ, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.542977033 (543h20m59.414060068s) [noise=true , address=145.79.227.13:4803 , piece_id=Y6J7IJAPDQYHALFN5O5ZQ67JURPO4EAUEUOXCUVI6E76TMFFZDLQ, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.542997411 (543h20m59.414080446s) [noise=true , address=45.91.116.13:4156 , piece_id=PGUEGBXASOPBNIODQ2G7SFBIYS25BH7EDRYPVPQ7VW4P4CFDI3TA, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.543381653 (543h20m59.414464688s) [noise=true , address=145.79.227.32:4835 , piece_id=TG7JI3BCO2Y2F2LHQAM6VEJYIKIENJ4PI6IYXA25K6K26XPUQ2KQ, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.543727081 (543h20m59.414810116s) [noise=true , address=145.79.227.6:4258 , piece_id=A3FIIPNCDDAM3O65WD5SFCVFBVLOFQHCH2U2ALP4I7KNPCEVFLLA, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.544084831 (543h20m59.415167877s) [noise=true , address=77.90.31.11:4554 , piece_id=U7YOB7RENTK43OCCNICZ7HZZFTSUTNXX5MLL2A3U6QNLXJXQEU5Q, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.544482139 (543h20m59.415565174s) [noise=true , address=45.91.116.12:4225 , piece_id=6F4QPKTMU5DJDRAGBV6XS3JXSA6T2Y5WV2KYGS36SSWPFAR37XYA, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.544826845 (543h20m59.41590988s) [noise=true , address=145.79.227.45:4230 , piece_id=36RAG2LPPCMAHMA7CGJPBN4JYOYIYIIGWJZCBKDGXW3JOTXL2RVQ, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.546119346 (543h20m59.417202391s) [noise=true , address=45.91.116.14:4189 , piece_id=KIPPHV5VAEGQ6PDIRQOGRP2OUCGMBTMDB4FU5CMY53DIQ34IOCTA, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.551669674 (543h20m59.42275272s) [noise=true , address=145.79.227.16:4909 , piece_id=OTDAYRDRJ3KND3GM7I3WMI3GLNSARWQZUFU2UZB5VSSQED6QUQGA, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:49:59.552830656 (543h20m59.423913691s) [noise=true , address=77.90.31.18:4808 , piece_id=532LOSBPEQNSBIMRW7FZM6DX5SBAILF3GDRNJGO7UIPBA5TG42FA, seg_pos={0 0}, upload=476491]: Uploading piece...
Dec 28 06:50:01.258365217 (543h21m1.129448292s) [upload=476491]: Next returned nil segment
Dec 28 06:50:01.258398810 (543h21m1.129481855s) [upload=476491]: Waiting for error group managing segments...
Dec 28 06:50:01.543372188 (543h21m1.414455233s) [noise=true , address=145.79.227.19:4078 , piece_id=I63FUO5ACRGWQGZRTSA7ELTWQT5ED5D3PEB2F7II5BGUKDY7GT6Q, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.565558916 (543h21m1.436641971s) [noise=true , address=145.79.227.13:4803 , piece_id=Y6J7IJAPDQYHALFN5O5ZQ67JURPO4EAUEUOXCUVI6E76TMFFZDLQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.576701373 (543h21m1.447784438s) [noise=true , address=145.79.227.6:4258 , piece_id=A3FIIPNCDDAM3O65WD5SFCVFBVLOFQHCH2U2ALP4I7KNPCEVFLLA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.579147681 (543h21m1.450230726s) [noise=true , address=145.79.227.38:4979 , piece_id=VEGJDYAE2ZNAQE2REEMMLRGTDLCSNLPJS7OYHUKJZBR5Q5IR5V4Q, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.585783266 (543h21m1.456866311s) [noise=true , address=145.79.227.32:4835 , piece_id=TG7JI3BCO2Y2F2LHQAM6VEJYIKIENJ4PI6IYXA25K6K26XPUQ2KQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.587092168 (543h21m1.458175213s) [noise=true , address=145.79.227.14:4839 , piece_id=HLWMDQCJYBRWGVQNJHR6X7JEMI5FXZCELVHJK2OX2XZDLKUNONPQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.588867348 (543h21m1.459950403s) [noise=true , address=145.79.227.46:4601 , piece_id=QJUNANFVIWS3RXHNSHQXV3QJLMLMVWSUJEXVRXEQ3IN4RF2BWGCA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.591090392 (543h21m1.462173437s) [noise=true , address=145.79.227.16:4909 , piece_id=OTDAYRDRJ3KND3GM7I3WMI3GLNSARWQZUFU2UZB5VSSQED6QUQGA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.591242731 (543h21m1.462325776s) [noise=true , address=145.79.227.45:4230 , piece_id=36RAG2LPPCMAHMA7CGJPBN4JYOYIYIIGWJZCBKDGXW3JOTXL2RVQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.718671558 (543h21m1.589754603s) [noise=true , address=77.90.31.10:4518 , piece_id=BL2KB2IMYCXC6YV64W4NUFTXMY3ZHQ54WRHQENSK5LVTCMAUVMJQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.786458726 (543h21m1.657541771s) [noise=true , address=45.91.116.12:4225 , piece_id=6F4QPKTMU5DJDRAGBV6XS3JXSA6T2Y5WV2KYGS36SSWPFAR37XYA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:01.954373975 (543h21m1.82545706s) [noise=true , address=45.91.116.11:4119 , piece_id=NGOLUFRHI3LNYUVX7W2NMFFZQNUGT7NTBZM5STOEDIKIX2KNB7OQ, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:02.201054377 (543h21m2.072137432s) [noise=true , address=77.90.31.11:4554 , piece_id=U7YOB7RENTK43OCCNICZ7HZZFTSUTNXX5MLL2A3U6QNLXJXQEU5Q, seg_pos={0 0}, upload=476491]: Done uploading piece. err: <nil>
Dec 28 06:50:02.201346554 (543h21m2.072429609s) [seg_pos={0 0}, upload=476491]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:50:02.201691010 (543h21m2.072774055s) [noise=true , address=77.90.31.18:4808 , piece_id=532LOSBPEQNSBIMRW7FZM6DX5SBAILF3GDRNJGO7UIPBA5TG42FA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: write tcp 188.166.225.170:44396->77.90.31.18:4808: use of closed network connection; write tcp 188.166.225.170:44396->77.90.31.18:4808: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44396->77.90.31.18:4808: use of closed network connection
Dec 28 06:50:02.201702451 (543h21m2.072785497s) [noise=true , address=77.90.31.18:4808 , piece_id=532LOSBPEQNSBIMRW7FZM6DX5SBAILF3GDRNJGO7UIPBA5TG42FA, seg_pos={0 0}, upload=476491]: Long tail cancellation due to optimal threshold
Dec 28 06:50:02.202552920 (543h21m2.073635966s) [noise=true , address=45.91.116.13:4156 , piece_id=PGUEGBXASOPBNIODQ2G7SFBIYS25BH7EDRYPVPQ7VW4P4CFDI3TA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 28 06:50:02.202565415 (543h21m2.07364846s) [noise=true , address=45.91.116.13:4156 , piece_id=PGUEGBXASOPBNIODQ2G7SFBIYS25BH7EDRYPVPQ7VW4P4CFDI3TA, seg_pos={0 0}, upload=476491]: Long tail cancellation due to optimal threshold
Dec 28 06:50:02.202783540 (543h21m2.073866595s) [noise=true , address=45.91.116.14:4189 , piece_id=KIPPHV5VAEGQ6PDIRQOGRP2OUCGMBTMDB4FU5CMY53DIQ34IOCTA, seg_pos={0 0}, upload=476491]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: read tcp 188.166.225.170:34022->45.91.116.14:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 06:50:02.202796154 (543h21m2.073879199s) [noise=true , address=45.91.116.14:4189 , piece_id=KIPPHV5VAEGQ6PDIRQOGRP2OUCGMBTMDB4FU5CMY53DIQ34IOCTA, seg_pos={0 0}, upload=476491]: Long tail cancellation due to optimal threshold
Dec 28 06:50:02.202823887 (543h21m2.073906932s) [seg_pos={0 0}, upload=476491]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:50:02.202945759 (543h21m2.074028804s) [upload=476491]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:50:02.202954987 (543h21m2.074038032s) [upload=476491]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:50:02.492076845 (543h21m2.36315989s) [upload=476491]: Done uploading segments...
Dec 28 06:50:02.492103696 (543h21m2.363186751s) [upload=476491]: Upload finished. err: <nil>
Duration: 30.003975462s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [X.XX.XXX..X..X.........X.XX..X....X.XX.........X.XXX.XX.....] duration:
5.766748248s trace:
https://jaeger.dev.storj.io/trace/68f38418308958ea
FAIL (2025-12-28 06:49:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA) context deadline exceeded
======LOGS======
17m22.768855999s) [upload=476437]: Done starting upload
Dec 28 06:46:22.897856913 (543h17m22.768939968s) [upload=476437]: Uploading segments...
Dec 28 06:46:22.898018331 (543h17m22.769101376s) [upload=476437]: Got next segment. Inline: false
Dec 28 06:46:22.898055381 (543h17m22.769138416s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:46:22.898059850 (543h17m22.769142895s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:46:23.146238344 (543h17m23.017321399s) [seg_pos={0 0}, upload=476437]: Begin upload segment...
Dec 28 06:46:23.146305923 (543h17m23.017388968s) [seg_pos={0 0}, upload=476437]: Done begin upload segment.
Dec 28 06:46:23.146346229 (543h17m23.017429274s) [noise=true , address=145.79.227.32:4835 , piece_id=YMQHHLZHOI7AAHYUHZUC32N2D2ZDLTMA25GOQANV6EHI52E5GTLA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.146387377 (543h17m23.017470413s) [noise=true , address=45.91.116.5:4298 , piece_id=HCXZH2OPQKMTC2TLJK3ZUIUQAWU56HGIDE5DPSIZJNMFJUSVFGYQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.147168195 (543h17m23.01825124s) [noise=true , address=145.79.227.46:4603 , piece_id=FRZ6AGFOXDBBDELJTHXERMLGFAOYRU6EQIT4Z67QYH7OROKNBFJQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.148178378 (543h17m23.019261413s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.148600843 (543h17m23.019683888s) [noise=true , address=45.91.116.13:4155 , piece_id=NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.149029719 (543h17m23.020112764s) [noise=true , address=77.90.31.16:4735 , piece_id=GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.149399624 (543h17m23.020482659s) [noise=true , address=45.91.116.6:4976 , piece_id=4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.149705747 (543h17m23.020788792s) [noise=true , address=145.79.227.18:4007 , piece_id=QAZIDUA73ACF6UDK7YURTYQHAY45VWMUMEMIFKSTHSGZYKDOVP5Q, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.151081216 (543h17m23.022164261s) [noise=true , address=145.79.227.41:4085 , piece_id=ILQAYUO5K2NEAYFDRHAGOLKKMERR2JQ7LPMSITYGSG5ZRTE5UHFA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.151102927 (543h17m23.022185972s) [noise=true , address=145.79.227.37:4567 , piece_id=JLZDGUNF4VEXY3XJET7XKBTDDAGISRQDDQD2BYEJIZB35EP5EUBQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.151161408 (543h17m23.022244453s) [noise=true , address=45.91.116.10:4083 , piece_id=OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.152308753 (543h17m23.023391798s) [noise=true , address=145.79.227.11:4729 , piece_id=2MJ42FM73EOIQ676TAWOERLDLKTEXMTUNZCWDRWCACE5PFFLQ5BQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.152614074 (543h17m23.023697119s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.153465506 (543h17m23.024548551s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.154192350 (543h17m23.025275385s) [noise=true , address=145.79.227.48:4302 , piece_id=U7WPLO4WQAQXTXPLQFYPMFNHDPXXLZW56Z7EBIMZNLZFJMGLVMKA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.154923322 (543h17m23.026006367s) [noise=true , address=145.79.227.4:4187 , piece_id=PX5LTAUH2ML3HFWM6VKHMILJ4ALA56JGEIWUVVJF7S3QAFDU3U5Q, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:24.875079193 (543h17m24.746162228s) [upload=476437]: Next returned nil segment
Dec 28 06:46:24.875097619 (543h17m24.746180664s) [upload=476437]: Waiting for error group managing segments...
Dec 28 06:46:25.162521353 (543h17m25.033604398s) [noise=true , address=145.79.227.32:4835 , piece_id=YMQHHLZHOI7AAHYUHZUC32N2D2ZDLTMA25GOQANV6EHI52E5GTLA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.181629498 (543h17m25.052712563s) [noise=true , address=145.79.227.48:4302 , piece_id=U7WPLO4WQAQXTXPLQFYPMFNHDPXXLZW56Z7EBIMZNLZFJMGLVMKA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.181886156 (543h17m25.052969191s) [noise=true , address=145.79.227.46:4603 , piece_id=FRZ6AGFOXDBBDELJTHXERMLGFAOYRU6EQIT4Z67QYH7OROKNBFJQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.189549649 (543h17m25.060632694s) [noise=true , address=145.79.227.4:4187 , piece_id=PX5LTAUH2ML3HFWM6VKHMILJ4ALA56JGEIWUVVJF7S3QAFDU3U5Q, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.318248013 (543h17m25.189331048s) [noise=true , address=145.79.227.11:4729 , piece_id=2MJ42FM73EOIQ676TAWOERLDLKTEXMTUNZCWDRWCACE5PFFLQ5BQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.318976890 (543h17m25.190059936s) [noise=true , address=145.79.227.37:4567 , piece_id=JLZDGUNF4VEXY3XJET7XKBTDDAGISRQDDQD2BYEJIZB35EP5EUBQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.340542916 (543h17m25.211625961s) [noise=true , address=145.79.227.18:4007 , piece_id=QAZIDUA73ACF6UDK7YURTYQHAY45VWMUMEMIFKSTHSGZYKDOVP5Q, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.344318485 (543h17m25.21540152s) [noise=true , address=77.90.31.16:4735 , piece_id=GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.389875724 (543h17m25.260958759s) [noise=true , address=45.91.116.10:4083 , piece_id=OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.418869634 (543h17m25.289952679s) [noise=true , address=45.91.116.6:4976 , piece_id=4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.466905512 (543h17m25.337988557s) [noise=true , address=45.91.116.13:4155 , piece_id=NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.518101047 (543h17m25.389184082s) [noise=true , address=145.79.227.41:4085 , piece_id=ILQAYUO5K2NEAYFDRHAGOLKKMERR2JQ7LPMSITYGSG5ZRTE5UHFA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.535957599 (543h17m25.407040644s) [noise=true , address=45.91.116.5:4298 , piece_id=HCXZH2OPQKMTC2TLJK3ZUIUQAWU56HGIDE5DPSIZJNMFJUSVFGYQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.535982507 (543h17m25.407065552s) [seg_pos={0 0}, upload=476437]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:46:25.536235087 (543h17m25.407318133s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF
Dec 28 06:46:25.536244976 (543h17m25.407328021s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold
Dec 28 06:46:25.536403298 (543h17m25.407486343s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection
Dec 28 06:46:25.536411383 (543h17m25.407494428s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold
Dec 28 06:46:25.536572661 (543h17m25.407655706s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:43800->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 06:46:25.536601586 (543h17m25.407684641s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold
Dec 28 06:46:25.536639398 (543h17m25.407722443s) [seg_pos={0 0}, upload=476437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:46:25.536742875 (543h17m25.40782592s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:46:25.536750008 (543h17m25.407833053s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:46:25.851474357 (543h17m25.722557402s) [upload=476437]: Done uploading segments...
Dec 28 06:46:25.851496479 (543h17m25.722579524s) [upload=476437]: Upload finished. err: <nil>
Duration: 5.766773465s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [X..XX.X...XX.....X................X.X.....X.X.........X.X...] duration:
5.001016023s trace:
https://jaeger.dev.storj.io/trace/2230181a8dada927
FAIL (2025-12-28 06:49:40 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001041851s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [.X..XX.X...XX.....X................X.X.....X.X.........X.X..] duration:
5.000546278s trace:
https://jaeger.dev.storj.io/trace/2230181a8dada927
FAIL (2025-12-28 06:49:12 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00057347s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.X.XX.XXX..X..X.........X.XX..X....X.XX.........X.XXX.XX....] duration:
5.938142682s trace:
https://jaeger.dev.storj.io/trace/68f38418308958ea
FAIL (2025-12-28 06:49:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA) context deadline exceeded
======LOGS======
17m22.768855999s) [upload=476437]: Done starting upload
Dec 28 06:46:22.897856913 (543h17m22.768939968s) [upload=476437]: Uploading segments...
Dec 28 06:46:22.898018331 (543h17m22.769101376s) [upload=476437]: Got next segment. Inline: false
Dec 28 06:46:22.898055381 (543h17m22.769138416s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:46:22.898059850 (543h17m22.769142895s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:46:23.146238344 (543h17m23.017321399s) [seg_pos={0 0}, upload=476437]: Begin upload segment...
Dec 28 06:46:23.146305923 (543h17m23.017388968s) [seg_pos={0 0}, upload=476437]: Done begin upload segment.
Dec 28 06:46:23.146346229 (543h17m23.017429274s) [noise=true , address=145.79.227.32:4835 , piece_id=YMQHHLZHOI7AAHYUHZUC32N2D2ZDLTMA25GOQANV6EHI52E5GTLA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.146387377 (543h17m23.017470413s) [noise=true , address=45.91.116.5:4298 , piece_id=HCXZH2OPQKMTC2TLJK3ZUIUQAWU56HGIDE5DPSIZJNMFJUSVFGYQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.147168195 (543h17m23.01825124s) [noise=true , address=145.79.227.46:4603 , piece_id=FRZ6AGFOXDBBDELJTHXERMLGFAOYRU6EQIT4Z67QYH7OROKNBFJQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.148178378 (543h17m23.019261413s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.148600843 (543h17m23.019683888s) [noise=true , address=45.91.116.13:4155 , piece_id=NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.149029719 (543h17m23.020112764s) [noise=true , address=77.90.31.16:4735 , piece_id=GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.149399624 (543h17m23.020482659s) [noise=true , address=45.91.116.6:4976 , piece_id=4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.149705747 (543h17m23.020788792s) [noise=true , address=145.79.227.18:4007 , piece_id=QAZIDUA73ACF6UDK7YURTYQHAY45VWMUMEMIFKSTHSGZYKDOVP5Q, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.151081216 (543h17m23.022164261s) [noise=true , address=145.79.227.41:4085 , piece_id=ILQAYUO5K2NEAYFDRHAGOLKKMERR2JQ7LPMSITYGSG5ZRTE5UHFA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.151102927 (543h17m23.022185972s) [noise=true , address=145.79.227.37:4567 , piece_id=JLZDGUNF4VEXY3XJET7XKBTDDAGISRQDDQD2BYEJIZB35EP5EUBQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.151161408 (543h17m23.022244453s) [noise=true , address=45.91.116.10:4083 , piece_id=OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.152308753 (543h17m23.023391798s) [noise=true , address=145.79.227.11:4729 , piece_id=2MJ42FM73EOIQ676TAWOERLDLKTEXMTUNZCWDRWCACE5PFFLQ5BQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.152614074 (543h17m23.023697119s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.153465506 (543h17m23.024548551s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.154192350 (543h17m23.025275385s) [noise=true , address=145.79.227.48:4302 , piece_id=U7WPLO4WQAQXTXPLQFYPMFNHDPXXLZW56Z7EBIMZNLZFJMGLVMKA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.154923322 (543h17m23.026006367s) [noise=true , address=145.79.227.4:4187 , piece_id=PX5LTAUH2ML3HFWM6VKHMILJ4ALA56JGEIWUVVJF7S3QAFDU3U5Q, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:24.875079193 (543h17m24.746162228s) [upload=476437]: Next returned nil segment
Dec 28 06:46:24.875097619 (543h17m24.746180664s) [upload=476437]: Waiting for error group managing segments...
Dec 28 06:46:25.162521353 (543h17m25.033604398s) [noise=true , address=145.79.227.32:4835 , piece_id=YMQHHLZHOI7AAHYUHZUC32N2D2ZDLTMA25GOQANV6EHI52E5GTLA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.181629498 (543h17m25.052712563s) [noise=true , address=145.79.227.48:4302 , piece_id=U7WPLO4WQAQXTXPLQFYPMFNHDPXXLZW56Z7EBIMZNLZFJMGLVMKA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.181886156 (543h17m25.052969191s) [noise=true , address=145.79.227.46:4603 , piece_id=FRZ6AGFOXDBBDELJTHXERMLGFAOYRU6EQIT4Z67QYH7OROKNBFJQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.189549649 (543h17m25.060632694s) [noise=true , address=145.79.227.4:4187 , piece_id=PX5LTAUH2ML3HFWM6VKHMILJ4ALA56JGEIWUVVJF7S3QAFDU3U5Q, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.318248013 (543h17m25.189331048s) [noise=true , address=145.79.227.11:4729 , piece_id=2MJ42FM73EOIQ676TAWOERLDLKTEXMTUNZCWDRWCACE5PFFLQ5BQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.318976890 (543h17m25.190059936s) [noise=true , address=145.79.227.37:4567 , piece_id=JLZDGUNF4VEXY3XJET7XKBTDDAGISRQDDQD2BYEJIZB35EP5EUBQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.340542916 (543h17m25.211625961s) [noise=true , address=145.79.227.18:4007 , piece_id=QAZIDUA73ACF6UDK7YURTYQHAY45VWMUMEMIFKSTHSGZYKDOVP5Q, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.344318485 (543h17m25.21540152s) [noise=true , address=77.90.31.16:4735 , piece_id=GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.389875724 (543h17m25.260958759s) [noise=true , address=45.91.116.10:4083 , piece_id=OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.418869634 (543h17m25.289952679s) [noise=true , address=45.91.116.6:4976 , piece_id=4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.466905512 (543h17m25.337988557s) [noise=true , address=45.91.116.13:4155 , piece_id=NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.518101047 (543h17m25.389184082s) [noise=true , address=145.79.227.41:4085 , piece_id=ILQAYUO5K2NEAYFDRHAGOLKKMERR2JQ7LPMSITYGSG5ZRTE5UHFA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.535957599 (543h17m25.407040644s) [noise=true , address=45.91.116.5:4298 , piece_id=HCXZH2OPQKMTC2TLJK3ZUIUQAWU56HGIDE5DPSIZJNMFJUSVFGYQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.535982507 (543h17m25.407065552s) [seg_pos={0 0}, upload=476437]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:46:25.536235087 (543h17m25.407318133s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF
Dec 28 06:46:25.536244976 (543h17m25.407328021s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold
Dec 28 06:46:25.536403298 (543h17m25.407486343s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection
Dec 28 06:46:25.536411383 (543h17m25.407494428s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold
Dec 28 06:46:25.536572661 (543h17m25.407655706s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:43800->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 06:46:25.536601586 (543h17m25.407684641s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold
Dec 28 06:46:25.536639398 (543h17m25.407722443s) [seg_pos={0 0}, upload=476437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:46:25.536742875 (543h17m25.40782592s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:46:25.536750008 (543h17m25.407833053s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:46:25.851474357 (543h17m25.722557402s) [upload=476437]: Done uploading segments...
Dec 28 06:46:25.851496479 (543h17m25.722579524s) [upload=476437]: Upload finished. err: <nil>
Duration: 5.938183329s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [..X..XX.X...XX.....X................X.X.....X.X.........X.X.] duration:
3.441231063s trace:
https://jaeger.dev.storj.io/trace/2230181a8dada927
FAIL (2025-12-28 06:48:24 UTC)
orbiter warning: context deadline exceeded
Duration: 3.441262053s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [..X.XX.XXX..X..X.........X.XX..X....X.XX.........X.XXX.XX...] duration:
9.849922848s trace:
https://jaeger.dev.storj.io/trace/68f38418308958ea
FAIL (2025-12-28 06:48:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA) context deadline exceeded
======LOGS======
17m22.768855999s) [upload=476437]: Done starting upload
Dec 28 06:46:22.897856913 (543h17m22.768939968s) [upload=476437]: Uploading segments...
Dec 28 06:46:22.898018331 (543h17m22.769101376s) [upload=476437]: Got next segment. Inline: false
Dec 28 06:46:22.898055381 (543h17m22.769138416s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:46:22.898059850 (543h17m22.769142895s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:46:23.146238344 (543h17m23.017321399s) [seg_pos={0 0}, upload=476437]: Begin upload segment...
Dec 28 06:46:23.146305923 (543h17m23.017388968s) [seg_pos={0 0}, upload=476437]: Done begin upload segment.
Dec 28 06:46:23.146346229 (543h17m23.017429274s) [noise=true , address=145.79.227.32:4835 , piece_id=YMQHHLZHOI7AAHYUHZUC32N2D2ZDLTMA25GOQANV6EHI52E5GTLA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.146387377 (543h17m23.017470413s) [noise=true , address=45.91.116.5:4298 , piece_id=HCXZH2OPQKMTC2TLJK3ZUIUQAWU56HGIDE5DPSIZJNMFJUSVFGYQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.147168195 (543h17m23.01825124s) [noise=true , address=145.79.227.46:4603 , piece_id=FRZ6AGFOXDBBDELJTHXERMLGFAOYRU6EQIT4Z67QYH7OROKNBFJQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.148178378 (543h17m23.019261413s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.148600843 (543h17m23.019683888s) [noise=true , address=45.91.116.13:4155 , piece_id=NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.149029719 (543h17m23.020112764s) [noise=true , address=77.90.31.16:4735 , piece_id=GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.149399624 (543h17m23.020482659s) [noise=true , address=45.91.116.6:4976 , piece_id=4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.149705747 (543h17m23.020788792s) [noise=true , address=145.79.227.18:4007 , piece_id=QAZIDUA73ACF6UDK7YURTYQHAY45VWMUMEMIFKSTHSGZYKDOVP5Q, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.151081216 (543h17m23.022164261s) [noise=true , address=145.79.227.41:4085 , piece_id=ILQAYUO5K2NEAYFDRHAGOLKKMERR2JQ7LPMSITYGSG5ZRTE5UHFA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.151102927 (543h17m23.022185972s) [noise=true , address=145.79.227.37:4567 , piece_id=JLZDGUNF4VEXY3XJET7XKBTDDAGISRQDDQD2BYEJIZB35EP5EUBQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.151161408 (543h17m23.022244453s) [noise=true , address=45.91.116.10:4083 , piece_id=OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.152308753 (543h17m23.023391798s) [noise=true , address=145.79.227.11:4729 , piece_id=2MJ42FM73EOIQ676TAWOERLDLKTEXMTUNZCWDRWCACE5PFFLQ5BQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.152614074 (543h17m23.023697119s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.153465506 (543h17m23.024548551s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.154192350 (543h17m23.025275385s) [noise=true , address=145.79.227.48:4302 , piece_id=U7WPLO4WQAQXTXPLQFYPMFNHDPXXLZW56Z7EBIMZNLZFJMGLVMKA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.154923322 (543h17m23.026006367s) [noise=true , address=145.79.227.4:4187 , piece_id=PX5LTAUH2ML3HFWM6VKHMILJ4ALA56JGEIWUVVJF7S3QAFDU3U5Q, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:24.875079193 (543h17m24.746162228s) [upload=476437]: Next returned nil segment
Dec 28 06:46:24.875097619 (543h17m24.746180664s) [upload=476437]: Waiting for error group managing segments...
Dec 28 06:46:25.162521353 (543h17m25.033604398s) [noise=true , address=145.79.227.32:4835 , piece_id=YMQHHLZHOI7AAHYUHZUC32N2D2ZDLTMA25GOQANV6EHI52E5GTLA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.181629498 (543h17m25.052712563s) [noise=true , address=145.79.227.48:4302 , piece_id=U7WPLO4WQAQXTXPLQFYPMFNHDPXXLZW56Z7EBIMZNLZFJMGLVMKA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.181886156 (543h17m25.052969191s) [noise=true , address=145.79.227.46:4603 , piece_id=FRZ6AGFOXDBBDELJTHXERMLGFAOYRU6EQIT4Z67QYH7OROKNBFJQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.189549649 (543h17m25.060632694s) [noise=true , address=145.79.227.4:4187 , piece_id=PX5LTAUH2ML3HFWM6VKHMILJ4ALA56JGEIWUVVJF7S3QAFDU3U5Q, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.318248013 (543h17m25.189331048s) [noise=true , address=145.79.227.11:4729 , piece_id=2MJ42FM73EOIQ676TAWOERLDLKTEXMTUNZCWDRWCACE5PFFLQ5BQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.318976890 (543h17m25.190059936s) [noise=true , address=145.79.227.37:4567 , piece_id=JLZDGUNF4VEXY3XJET7XKBTDDAGISRQDDQD2BYEJIZB35EP5EUBQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.340542916 (543h17m25.211625961s) [noise=true , address=145.79.227.18:4007 , piece_id=QAZIDUA73ACF6UDK7YURTYQHAY45VWMUMEMIFKSTHSGZYKDOVP5Q, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.344318485 (543h17m25.21540152s) [noise=true , address=77.90.31.16:4735 , piece_id=GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.389875724 (543h17m25.260958759s) [noise=true , address=45.91.116.10:4083 , piece_id=OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.418869634 (543h17m25.289952679s) [noise=true , address=45.91.116.6:4976 , piece_id=4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.466905512 (543h17m25.337988557s) [noise=true , address=45.91.116.13:4155 , piece_id=NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.518101047 (543h17m25.389184082s) [noise=true , address=145.79.227.41:4085 , piece_id=ILQAYUO5K2NEAYFDRHAGOLKKMERR2JQ7LPMSITYGSG5ZRTE5UHFA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.535957599 (543h17m25.407040644s) [noise=true , address=45.91.116.5:4298 , piece_id=HCXZH2OPQKMTC2TLJK3ZUIUQAWU56HGIDE5DPSIZJNMFJUSVFGYQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.535982507 (543h17m25.407065552s) [seg_pos={0 0}, upload=476437]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:46:25.536235087 (543h17m25.407318133s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF
Dec 28 06:46:25.536244976 (543h17m25.407328021s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold
Dec 28 06:46:25.536403298 (543h17m25.407486343s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection
Dec 28 06:46:25.536411383 (543h17m25.407494428s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold
Dec 28 06:46:25.536572661 (543h17m25.407655706s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:43800->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 06:46:25.536601586 (543h17m25.407684641s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold
Dec 28 06:46:25.536639398 (543h17m25.407722443s) [seg_pos={0 0}, upload=476437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:46:25.536742875 (543h17m25.40782592s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:46:25.536750008 (543h17m25.407833053s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:46:25.851474357 (543h17m25.722557402s) [upload=476437]: Done uploading segments...
Dec 28 06:46:25.851496479 (543h17m25.722579524s) [upload=476437]: Upload finished. err: <nil>
Duration: 9.849951464s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [X..X.XX.XXX..X..X.........X.XX..X....X.XX.........X.XXX.XX..] duration:
5.741629728s trace:
https://jaeger.dev.storj.io/trace/68f38418308958ea
FAIL (2025-12-28 06:47:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA) context deadline exceeded
======LOGS======
17m22.768855999s) [upload=476437]: Done starting upload
Dec 28 06:46:22.897856913 (543h17m22.768939968s) [upload=476437]: Uploading segments...
Dec 28 06:46:22.898018331 (543h17m22.769101376s) [upload=476437]: Got next segment. Inline: false
Dec 28 06:46:22.898055381 (543h17m22.769138416s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:46:22.898059850 (543h17m22.769142895s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:46:23.146238344 (543h17m23.017321399s) [seg_pos={0 0}, upload=476437]: Begin upload segment...
Dec 28 06:46:23.146305923 (543h17m23.017388968s) [seg_pos={0 0}, upload=476437]: Done begin upload segment.
Dec 28 06:46:23.146346229 (543h17m23.017429274s) [noise=true , address=145.79.227.32:4835 , piece_id=YMQHHLZHOI7AAHYUHZUC32N2D2ZDLTMA25GOQANV6EHI52E5GTLA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.146387377 (543h17m23.017470413s) [noise=true , address=45.91.116.5:4298 , piece_id=HCXZH2OPQKMTC2TLJK3ZUIUQAWU56HGIDE5DPSIZJNMFJUSVFGYQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.147168195 (543h17m23.01825124s) [noise=true , address=145.79.227.46:4603 , piece_id=FRZ6AGFOXDBBDELJTHXERMLGFAOYRU6EQIT4Z67QYH7OROKNBFJQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.148178378 (543h17m23.019261413s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.148600843 (543h17m23.019683888s) [noise=true , address=45.91.116.13:4155 , piece_id=NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.149029719 (543h17m23.020112764s) [noise=true , address=77.90.31.16:4735 , piece_id=GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.149399624 (543h17m23.020482659s) [noise=true , address=45.91.116.6:4976 , piece_id=4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.149705747 (543h17m23.020788792s) [noise=true , address=145.79.227.18:4007 , piece_id=QAZIDUA73ACF6UDK7YURTYQHAY45VWMUMEMIFKSTHSGZYKDOVP5Q, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.151081216 (543h17m23.022164261s) [noise=true , address=145.79.227.41:4085 , piece_id=ILQAYUO5K2NEAYFDRHAGOLKKMERR2JQ7LPMSITYGSG5ZRTE5UHFA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.151102927 (543h17m23.022185972s) [noise=true , address=145.79.227.37:4567 , piece_id=JLZDGUNF4VEXY3XJET7XKBTDDAGISRQDDQD2BYEJIZB35EP5EUBQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.151161408 (543h17m23.022244453s) [noise=true , address=45.91.116.10:4083 , piece_id=OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.152308753 (543h17m23.023391798s) [noise=true , address=145.79.227.11:4729 , piece_id=2MJ42FM73EOIQ676TAWOERLDLKTEXMTUNZCWDRWCACE5PFFLQ5BQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.152614074 (543h17m23.023697119s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.153465506 (543h17m23.024548551s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.154192350 (543h17m23.025275385s) [noise=true , address=145.79.227.48:4302 , piece_id=U7WPLO4WQAQXTXPLQFYPMFNHDPXXLZW56Z7EBIMZNLZFJMGLVMKA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.154923322 (543h17m23.026006367s) [noise=true , address=145.79.227.4:4187 , piece_id=PX5LTAUH2ML3HFWM6VKHMILJ4ALA56JGEIWUVVJF7S3QAFDU3U5Q, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:24.875079193 (543h17m24.746162228s) [upload=476437]: Next returned nil segment
Dec 28 06:46:24.875097619 (543h17m24.746180664s) [upload=476437]: Waiting for error group managing segments...
Dec 28 06:46:25.162521353 (543h17m25.033604398s) [noise=true , address=145.79.227.32:4835 , piece_id=YMQHHLZHOI7AAHYUHZUC32N2D2ZDLTMA25GOQANV6EHI52E5GTLA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.181629498 (543h17m25.052712563s) [noise=true , address=145.79.227.48:4302 , piece_id=U7WPLO4WQAQXTXPLQFYPMFNHDPXXLZW56Z7EBIMZNLZFJMGLVMKA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.181886156 (543h17m25.052969191s) [noise=true , address=145.79.227.46:4603 , piece_id=FRZ6AGFOXDBBDELJTHXERMLGFAOYRU6EQIT4Z67QYH7OROKNBFJQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.189549649 (543h17m25.060632694s) [noise=true , address=145.79.227.4:4187 , piece_id=PX5LTAUH2ML3HFWM6VKHMILJ4ALA56JGEIWUVVJF7S3QAFDU3U5Q, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.318248013 (543h17m25.189331048s) [noise=true , address=145.79.227.11:4729 , piece_id=2MJ42FM73EOIQ676TAWOERLDLKTEXMTUNZCWDRWCACE5PFFLQ5BQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.318976890 (543h17m25.190059936s) [noise=true , address=145.79.227.37:4567 , piece_id=JLZDGUNF4VEXY3XJET7XKBTDDAGISRQDDQD2BYEJIZB35EP5EUBQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.340542916 (543h17m25.211625961s) [noise=true , address=145.79.227.18:4007 , piece_id=QAZIDUA73ACF6UDK7YURTYQHAY45VWMUMEMIFKSTHSGZYKDOVP5Q, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.344318485 (543h17m25.21540152s) [noise=true , address=77.90.31.16:4735 , piece_id=GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.389875724 (543h17m25.260958759s) [noise=true , address=45.91.116.10:4083 , piece_id=OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.418869634 (543h17m25.289952679s) [noise=true , address=45.91.116.6:4976 , piece_id=4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.466905512 (543h17m25.337988557s) [noise=true , address=45.91.116.13:4155 , piece_id=NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.518101047 (543h17m25.389184082s) [noise=true , address=145.79.227.41:4085 , piece_id=ILQAYUO5K2NEAYFDRHAGOLKKMERR2JQ7LPMSITYGSG5ZRTE5UHFA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.535957599 (543h17m25.407040644s) [noise=true , address=45.91.116.5:4298 , piece_id=HCXZH2OPQKMTC2TLJK3ZUIUQAWU56HGIDE5DPSIZJNMFJUSVFGYQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.535982507 (543h17m25.407065552s) [seg_pos={0 0}, upload=476437]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:46:25.536235087 (543h17m25.407318133s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF
Dec 28 06:46:25.536244976 (543h17m25.407328021s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold
Dec 28 06:46:25.536403298 (543h17m25.407486343s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection
Dec 28 06:46:25.536411383 (543h17m25.407494428s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold
Dec 28 06:46:25.536572661 (543h17m25.407655706s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:43800->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 06:46:25.536601586 (543h17m25.407684641s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold
Dec 28 06:46:25.536639398 (543h17m25.407722443s) [seg_pos={0 0}, upload=476437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:46:25.536742875 (543h17m25.40782592s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:46:25.536750008 (543h17m25.407833053s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:46:25.851474357 (543h17m25.722557402s) [upload=476437]: Done uploading segments...
Dec 28 06:46:25.851496479 (543h17m25.722579524s) [upload=476437]: Upload finished. err: <nil>
Duration: 5.741663261s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [.X..X.XX.XXX..X..X.........X.XX..X....X.XX.........X.XXX.XX.] duration:
27.194348564s trace:
https://jaeger.dev.storj.io/trace/68f38418308958ea
FAIL (2025-12-28 06:47:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA) context deadline exceeded
======LOGS======
17m22.768855999s) [upload=476437]: Done starting upload
Dec 28 06:46:22.897856913 (543h17m22.768939968s) [upload=476437]: Uploading segments...
Dec 28 06:46:22.898018331 (543h17m22.769101376s) [upload=476437]: Got next segment. Inline: false
Dec 28 06:46:22.898055381 (543h17m22.769138416s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:46:22.898059850 (543h17m22.769142895s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:46:23.146238344 (543h17m23.017321399s) [seg_pos={0 0}, upload=476437]: Begin upload segment...
Dec 28 06:46:23.146305923 (543h17m23.017388968s) [seg_pos={0 0}, upload=476437]: Done begin upload segment.
Dec 28 06:46:23.146346229 (543h17m23.017429274s) [noise=true , address=145.79.227.32:4835 , piece_id=YMQHHLZHOI7AAHYUHZUC32N2D2ZDLTMA25GOQANV6EHI52E5GTLA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.146387377 (543h17m23.017470413s) [noise=true , address=45.91.116.5:4298 , piece_id=HCXZH2OPQKMTC2TLJK3ZUIUQAWU56HGIDE5DPSIZJNMFJUSVFGYQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.147168195 (543h17m23.01825124s) [noise=true , address=145.79.227.46:4603 , piece_id=FRZ6AGFOXDBBDELJTHXERMLGFAOYRU6EQIT4Z67QYH7OROKNBFJQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.148178378 (543h17m23.019261413s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.148600843 (543h17m23.019683888s) [noise=true , address=45.91.116.13:4155 , piece_id=NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.149029719 (543h17m23.020112764s) [noise=true , address=77.90.31.16:4735 , piece_id=GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.149399624 (543h17m23.020482659s) [noise=true , address=45.91.116.6:4976 , piece_id=4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.149705747 (543h17m23.020788792s) [noise=true , address=145.79.227.18:4007 , piece_id=QAZIDUA73ACF6UDK7YURTYQHAY45VWMUMEMIFKSTHSGZYKDOVP5Q, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.151081216 (543h17m23.022164261s) [noise=true , address=145.79.227.41:4085 , piece_id=ILQAYUO5K2NEAYFDRHAGOLKKMERR2JQ7LPMSITYGSG5ZRTE5UHFA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.151102927 (543h17m23.022185972s) [noise=true , address=145.79.227.37:4567 , piece_id=JLZDGUNF4VEXY3XJET7XKBTDDAGISRQDDQD2BYEJIZB35EP5EUBQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.151161408 (543h17m23.022244453s) [noise=true , address=45.91.116.10:4083 , piece_id=OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.152308753 (543h17m23.023391798s) [noise=true , address=145.79.227.11:4729 , piece_id=2MJ42FM73EOIQ676TAWOERLDLKTEXMTUNZCWDRWCACE5PFFLQ5BQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.152614074 (543h17m23.023697119s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.153465506 (543h17m23.024548551s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.154192350 (543h17m23.025275385s) [noise=true , address=145.79.227.48:4302 , piece_id=U7WPLO4WQAQXTXPLQFYPMFNHDPXXLZW56Z7EBIMZNLZFJMGLVMKA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.154923322 (543h17m23.026006367s) [noise=true , address=145.79.227.4:4187 , piece_id=PX5LTAUH2ML3HFWM6VKHMILJ4ALA56JGEIWUVVJF7S3QAFDU3U5Q, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:24.875079193 (543h17m24.746162228s) [upload=476437]: Next returned nil segment
Dec 28 06:46:24.875097619 (543h17m24.746180664s) [upload=476437]: Waiting for error group managing segments...
Dec 28 06:46:25.162521353 (543h17m25.033604398s) [noise=true , address=145.79.227.32:4835 , piece_id=YMQHHLZHOI7AAHYUHZUC32N2D2ZDLTMA25GOQANV6EHI52E5GTLA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.181629498 (543h17m25.052712563s) [noise=true , address=145.79.227.48:4302 , piece_id=U7WPLO4WQAQXTXPLQFYPMFNHDPXXLZW56Z7EBIMZNLZFJMGLVMKA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.181886156 (543h17m25.052969191s) [noise=true , address=145.79.227.46:4603 , piece_id=FRZ6AGFOXDBBDELJTHXERMLGFAOYRU6EQIT4Z67QYH7OROKNBFJQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.189549649 (543h17m25.060632694s) [noise=true , address=145.79.227.4:4187 , piece_id=PX5LTAUH2ML3HFWM6VKHMILJ4ALA56JGEIWUVVJF7S3QAFDU3U5Q, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.318248013 (543h17m25.189331048s) [noise=true , address=145.79.227.11:4729 , piece_id=2MJ42FM73EOIQ676TAWOERLDLKTEXMTUNZCWDRWCACE5PFFLQ5BQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.318976890 (543h17m25.190059936s) [noise=true , address=145.79.227.37:4567 , piece_id=JLZDGUNF4VEXY3XJET7XKBTDDAGISRQDDQD2BYEJIZB35EP5EUBQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.340542916 (543h17m25.211625961s) [noise=true , address=145.79.227.18:4007 , piece_id=QAZIDUA73ACF6UDK7YURTYQHAY45VWMUMEMIFKSTHSGZYKDOVP5Q, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.344318485 (543h17m25.21540152s) [noise=true , address=77.90.31.16:4735 , piece_id=GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.389875724 (543h17m25.260958759s) [noise=true , address=45.91.116.10:4083 , piece_id=OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.418869634 (543h17m25.289952679s) [noise=true , address=45.91.116.6:4976 , piece_id=4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.466905512 (543h17m25.337988557s) [noise=true , address=45.91.116.13:4155 , piece_id=NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.518101047 (543h17m25.389184082s) [noise=true , address=145.79.227.41:4085 , piece_id=ILQAYUO5K2NEAYFDRHAGOLKKMERR2JQ7LPMSITYGSG5ZRTE5UHFA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.535957599 (543h17m25.407040644s) [noise=true , address=45.91.116.5:4298 , piece_id=HCXZH2OPQKMTC2TLJK3ZUIUQAWU56HGIDE5DPSIZJNMFJUSVFGYQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.535982507 (543h17m25.407065552s) [seg_pos={0 0}, upload=476437]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:46:25.536235087 (543h17m25.407318133s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF
Dec 28 06:46:25.536244976 (543h17m25.407328021s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold
Dec 28 06:46:25.536403298 (543h17m25.407486343s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection
Dec 28 06:46:25.536411383 (543h17m25.407494428s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold
Dec 28 06:46:25.536572661 (543h17m25.407655706s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:43800->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 06:46:25.536601586 (543h17m25.407684641s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold
Dec 28 06:46:25.536639398 (543h17m25.407722443s) [seg_pos={0 0}, upload=476437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:46:25.536742875 (543h17m25.40782592s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:46:25.536750008 (543h17m25.407833053s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:46:25.851474357 (543h17m25.722557402s) [upload=476437]: Done uploading segments...
Dec 28 06:46:25.851496479 (543h17m25.722579524s) [upload=476437]: Upload finished. err: <nil>
Duration: 27.194373621s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [...X..XX.X...XX.....X................X.X.....X.X.........X.X] duration:
5.001192668s trace:
https://jaeger.dev.storj.io/trace/2230181a8dada927
FAIL (2025-12-28 06:47:21 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00121987s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [..X..X.XX.XXX..X..X.........X.XX..X....X.XX.........X.XXX.XX] duration:
30.004716052s trace:
https://jaeger.dev.storj.io/trace/68f38418308958ea
FAIL (2025-12-28 06:46:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, Piece ID: NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA) context deadline exceeded
======LOGS======
17m22.768855999s) [upload=476437]: Done starting upload
Dec 28 06:46:22.897856913 (543h17m22.768939968s) [upload=476437]: Uploading segments...
Dec 28 06:46:22.898018331 (543h17m22.769101376s) [upload=476437]: Got next segment. Inline: false
Dec 28 06:46:22.898055381 (543h17m22.769138416s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:46:22.898059850 (543h17m22.769142895s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:46:23.146238344 (543h17m23.017321399s) [seg_pos={0 0}, upload=476437]: Begin upload segment...
Dec 28 06:46:23.146305923 (543h17m23.017388968s) [seg_pos={0 0}, upload=476437]: Done begin upload segment.
Dec 28 06:46:23.146346229 (543h17m23.017429274s) [noise=true , address=145.79.227.32:4835 , piece_id=YMQHHLZHOI7AAHYUHZUC32N2D2ZDLTMA25GOQANV6EHI52E5GTLA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.146387377 (543h17m23.017470413s) [noise=true , address=45.91.116.5:4298 , piece_id=HCXZH2OPQKMTC2TLJK3ZUIUQAWU56HGIDE5DPSIZJNMFJUSVFGYQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.147168195 (543h17m23.01825124s) [noise=true , address=145.79.227.46:4603 , piece_id=FRZ6AGFOXDBBDELJTHXERMLGFAOYRU6EQIT4Z67QYH7OROKNBFJQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.148178378 (543h17m23.019261413s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.148600843 (543h17m23.019683888s) [noise=true , address=45.91.116.13:4155 , piece_id=NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.149029719 (543h17m23.020112764s) [noise=true , address=77.90.31.16:4735 , piece_id=GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.149399624 (543h17m23.020482659s) [noise=true , address=45.91.116.6:4976 , piece_id=4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.149705747 (543h17m23.020788792s) [noise=true , address=145.79.227.18:4007 , piece_id=QAZIDUA73ACF6UDK7YURTYQHAY45VWMUMEMIFKSTHSGZYKDOVP5Q, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.151081216 (543h17m23.022164261s) [noise=true , address=145.79.227.41:4085 , piece_id=ILQAYUO5K2NEAYFDRHAGOLKKMERR2JQ7LPMSITYGSG5ZRTE5UHFA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.151102927 (543h17m23.022185972s) [noise=true , address=145.79.227.37:4567 , piece_id=JLZDGUNF4VEXY3XJET7XKBTDDAGISRQDDQD2BYEJIZB35EP5EUBQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.151161408 (543h17m23.022244453s) [noise=true , address=45.91.116.10:4083 , piece_id=OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.152308753 (543h17m23.023391798s) [noise=true , address=145.79.227.11:4729 , piece_id=2MJ42FM73EOIQ676TAWOERLDLKTEXMTUNZCWDRWCACE5PFFLQ5BQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.152614074 (543h17m23.023697119s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.153465506 (543h17m23.024548551s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.154192350 (543h17m23.025275385s) [noise=true , address=145.79.227.48:4302 , piece_id=U7WPLO4WQAQXTXPLQFYPMFNHDPXXLZW56Z7EBIMZNLZFJMGLVMKA, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:23.154923322 (543h17m23.026006367s) [noise=true , address=145.79.227.4:4187 , piece_id=PX5LTAUH2ML3HFWM6VKHMILJ4ALA56JGEIWUVVJF7S3QAFDU3U5Q, seg_pos={0 0}, upload=476437]: Uploading piece...
Dec 28 06:46:24.875079193 (543h17m24.746162228s) [upload=476437]: Next returned nil segment
Dec 28 06:46:24.875097619 (543h17m24.746180664s) [upload=476437]: Waiting for error group managing segments...
Dec 28 06:46:25.162521353 (543h17m25.033604398s) [noise=true , address=145.79.227.32:4835 , piece_id=YMQHHLZHOI7AAHYUHZUC32N2D2ZDLTMA25GOQANV6EHI52E5GTLA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.181629498 (543h17m25.052712563s) [noise=true , address=145.79.227.48:4302 , piece_id=U7WPLO4WQAQXTXPLQFYPMFNHDPXXLZW56Z7EBIMZNLZFJMGLVMKA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.181886156 (543h17m25.052969191s) [noise=true , address=145.79.227.46:4603 , piece_id=FRZ6AGFOXDBBDELJTHXERMLGFAOYRU6EQIT4Z67QYH7OROKNBFJQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.189549649 (543h17m25.060632694s) [noise=true , address=145.79.227.4:4187 , piece_id=PX5LTAUH2ML3HFWM6VKHMILJ4ALA56JGEIWUVVJF7S3QAFDU3U5Q, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.318248013 (543h17m25.189331048s) [noise=true , address=145.79.227.11:4729 , piece_id=2MJ42FM73EOIQ676TAWOERLDLKTEXMTUNZCWDRWCACE5PFFLQ5BQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.318976890 (543h17m25.190059936s) [noise=true , address=145.79.227.37:4567 , piece_id=JLZDGUNF4VEXY3XJET7XKBTDDAGISRQDDQD2BYEJIZB35EP5EUBQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.340542916 (543h17m25.211625961s) [noise=true , address=145.79.227.18:4007 , piece_id=QAZIDUA73ACF6UDK7YURTYQHAY45VWMUMEMIFKSTHSGZYKDOVP5Q, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.344318485 (543h17m25.21540152s) [noise=true , address=77.90.31.16:4735 , piece_id=GOFOB6Z67RHERUIWWFWTPXQBEX7Q7ZW4RHIWMLPH66J256YIMMKA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.389875724 (543h17m25.260958759s) [noise=true , address=45.91.116.10:4083 , piece_id=OK3Z3JR6L5WVR4OVNHX5Y6ELCLQQENGLRMNJ3BPEC6AKYK7ZO7JA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.418869634 (543h17m25.289952679s) [noise=true , address=45.91.116.6:4976 , piece_id=4AGOAOFGZE7MQ5VZ46UOPIMDZT4CLQVOJIIX6ZPTP2PYIR2S6PUA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.466905512 (543h17m25.337988557s) [noise=true , address=45.91.116.13:4155 , piece_id=NO57FUQTY647FCEEINKGJRE2UVHUTC2D4JESMAPNQDCSXQIXEQQA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.518101047 (543h17m25.389184082s) [noise=true , address=145.79.227.41:4085 , piece_id=ILQAYUO5K2NEAYFDRHAGOLKKMERR2JQ7LPMSITYGSG5ZRTE5UHFA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.535957599 (543h17m25.407040644s) [noise=true , address=45.91.116.5:4298 , piece_id=HCXZH2OPQKMTC2TLJK3ZUIUQAWU56HGIDE5DPSIZJNMFJUSVFGYQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: <nil>
Dec 28 06:46:25.535982507 (543h17m25.407065552s) [seg_pos={0 0}, upload=476437]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:46:25.536235087 (543h17m25.407318133s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF
Dec 28 06:46:25.536244976 (543h17m25.407328021s) [noise=true , address=45.91.116.8:4046 , piece_id=GGFPJKARFC3HYJJOSOLYT3ZNSBUFZZSHTTHSQTBL2A4OO3CUMVZA, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold
Dec 28 06:46:25.536403298 (543h17m25.407486343s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection; write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56042->145.79.227.14:4840: use of closed network connection
Dec 28 06:46:25.536411383 (543h17m25.407494428s) [noise=true , address=145.79.227.14:4840 , piece_id=W5SXTKO5GPXZTXNIZB2V5WWNAP3KHWQB2OTALYRT6IMVEWR4O6SQ, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold
Dec 28 06:46:25.536572661 (543h17m25.407655706s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: read tcp 188.166.225.170:43800->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 06:46:25.536601586 (543h17m25.407684641s) [noise=true , address=45.91.116.7:4010 , piece_id=KDTP5P2AAYOHKOLK6BPFNGUIKNHPUTVDKK7TOURFY3LPWMTW7MZQ, seg_pos={0 0}, upload=476437]: Long tail cancellation due to optimal threshold
Dec 28 06:46:25.536639398 (543h17m25.407722443s) [seg_pos={0 0}, upload=476437]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:46:25.536742875 (543h17m25.40782592s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:46:25.536750008 (543h17m25.407833053s) [upload=476437]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:46:25.851474357 (543h17m25.722557402s) [upload=476437]: Done uploading segments...
Dec 28 06:46:25.851496479 (543h17m25.722579524s) [upload=476437]: Upload finished. err: <nil>
Duration: 30.004733396s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [...X..X.XX.XXX..X..X.........X.XX..X....X.XX.........X.XXX.X] duration:
30.003477443s trace:
https://jaeger.dev.storj.io/trace/529be33d1cffa5fa
FAIL (2025-12-28 06:46:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: UQHLCLK6KF7JDKZN7MDAAUQWV6I3UUSZAFU73Z2LLAAMERLLNQRA) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: UA6DQQODGIBYTNFLAMINYZUZHQLGU27CFTLIVLOKNKWCBCSABPYQ) context deadline exceeded
======LOGS======
:52.685128469 (543h16m52.556211514s) [upload=476427]: Done starting upload
Dec 28 06:45:52.685291460 (543h16m52.556374495s) [upload=476427]: Uploading segments...
Dec 28 06:45:52.685343999 (543h16m52.556427045s) [upload=476427]: Got next segment. Inline: false
Dec 28 06:45:52.685366693 (543h16m52.556449728s) [upload=476427]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:45:52.685370811 (543h16m52.556453846s) [upload=476427]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:45:52.934816007 (543h16m52.805899052s) [seg_pos={0 0}, upload=476427]: Begin upload segment...
Dec 28 06:45:52.934875821 (543h16m52.805958866s) [seg_pos={0 0}, upload=476427]: Done begin upload segment.
Dec 28 06:45:52.934925645 (543h16m52.80600868s) [noise=true , address=45.91.116.10:4083 , piece_id=UQHLCLK6KF7JDKZN7MDAAUQWV6I3UUSZAFU73Z2LLAAMERLLNQRA, seg_pos={0 0}, upload=476427]: Uploading piece...
Dec 28 06:45:52.934944131 (543h16m52.806027166s) [noise=true , address=77.90.31.14:4661 , piece_id=HICPDHHKVEYA4WR7DQCBUHVXT6Q4VG7M6LZF53ZUOVMA2AQR7MEQ, seg_pos={0 0}, upload=476427]: Uploading piece...
Dec 28 06:45:52.935083106 (543h16m52.806166151s) [noise=true , address=145.79.227.4:4187 , piece_id=4DPDKL647IAL47ARF2JZ2KRIULRQHD52CDWIF2W24YE2F2KRRRVQ, seg_pos={0 0}, upload=476427]: Uploading piece...
Dec 28 06:45:52.935577377 (543h16m52.806660422s) [noise=true , address=45.91.116.7:4012 , piece_id=UA6DQQODGIBYTNFLAMINYZUZHQLGU27CFTLIVLOKNKWCBCSABPYQ, seg_pos={0 0}, upload=476427]: Uploading piece...
Dec 28 06:45:52.935794840 (543h16m52.806877886s) [noise=true , address=145.79.227.51:4409 , piece_id=JFKX6FZG5MH6JMZKC5SHWVZJAJUAYHXIULSXLBCP6RNIWM66KJ2Q, seg_pos={0 0}, upload=476427]: Uploading piece...
Dec 28 06:45:52.936030930 (543h16m52.807113975s) [noise=true , address=145.79.227.39:4015 , piece_id=EKRM55RLEPDPWPCLDHOLQZMUAF7ZEPBLSTR2JV3OOIK5C3WRGJCQ, seg_pos={0 0}, upload=476427]: Uploading piece...
Dec 28 06:45:52.936291517 (543h16m52.807374562s) [noise=true , address=145.79.227.29:4941 , piece_id=QHADVHLW2ZT65NKXOZDSY5VCA7LHVMTTKUMVXUZY7UDOG4DTXBAQ, seg_pos={0 0}, upload=476427]: Uploading piece...
Dec 28 06:45:52.936463314 (543h16m52.807546359s) [noise=true , address=145.79.227.15:4875 , piece_id=4LPNBHB52QJYWSCFX62ENK4JVXO7JHM6ABVTWCWJALYR4AUKLM3A, seg_pos={0 0}, upload=476427]: Uploading piece...
Dec 28 06:45:52.936886169 (543h16m52.807969214s) [noise=true , address=145.79.227.2:4115 , piece_id=I343NB67NV6ZWPK5OKL4GYOTWD7T576HLH7IMHCP6OCHEPFTD4DQ, seg_pos={0 0}, upload=476427]: Uploading piece...
Dec 28 06:45:52.936934862 (543h16m52.808017907s) [noise=true , address=145.79.227.19:4079 , piece_id=NCHK5HX2KDH6FJTPMEIWX6T6WJLTIPKLTRSJQJYA7DHKZ63ATANQ, seg_pos={0 0}, upload=476427]: Uploading piece...
Dec 28 06:45:52.937391571 (543h16m52.808474616s) [noise=true , address=145.79.227.42:4123 , piece_id=WNF2JBCTXYTHP3LOCMSPFPZVDLDTOJ7MAM4TPOAMKKS2WKYVLD7A, seg_pos={0 0}, upload=476427]: Uploading piece...
Dec 28 06:45:52.937405909 (543h16m52.808488944s) [noise=true , address=77.90.31.18:4805 , piece_id=JHRH2QGRYW2PXJVJS56NJSMJYUIMZR2GWDS5MB6Q47ENC5IFJB4A, seg_pos={0 0}, upload=476427]: Uploading piece...
Dec 28 06:45:52.937471334 (543h16m52.808554379s) [noise=true , address=145.79.227.47:4267 , piece_id=M6QMLGZDNBJKRCYJDT5CG32BMESHXZLHMJY5EQ6JGFNI43NFO2DQ, seg_pos={0 0}, upload=476427]: Uploading piece...
Dec 28 06:45:52.937922302 (543h16m52.809005347s) [noise=true , address=45.91.116.4:4940 , piece_id=PJS2P36CDTPCJ6ZPVDQKMCJNV6AZ2NBB7MBCZWX2EKGWC7RJKPCA, seg_pos={0 0}, upload=476427]: Uploading piece...
Dec 28 06:45:52.940351157 (543h16m52.811434202s) [noise=true , address=145.79.227.16:4911 , piece_id=O7LIGIMP42EAVQRWKO4VYUFE33EKRTDKUYDOYXCQX7C33ZYTMKPQ, seg_pos={0 0}, upload=476427]: Uploading piece...
Dec 28 06:45:52.941083591 (543h16m52.812166626s) [noise=true , address=145.79.227.6:4258 , piece_id=QTU5PW4EM4TKEUWA3UNI36PLBBVBJS7P3KGIND6VW43CHVR2GSBA, seg_pos={0 0}, upload=476427]: Uploading piece...
Dec 28 06:45:54.840958208 (543h16m54.712041253s) [upload=476427]: Next returned nil segment
Dec 28 06:45:54.840976764 (543h16m54.712059809s) [upload=476427]: Waiting for error group managing segments...
Dec 28 06:45:55.163677321 (543h16m55.034760367s) [noise=true , address=45.91.116.4:4940 , piece_id=PJS2P36CDTPCJ6ZPVDQKMCJNV6AZ2NBB7MBCZWX2EKGWC7RJKPCA, seg_pos={0 0}, upload=476427]: Done uploading piece. err: <nil>
Dec 28 06:45:55.202209473 (543h16m55.073292518s) [noise=true , address=77.90.31.14:4661 , piece_id=HICPDHHKVEYA4WR7DQCBUHVXT6Q4VG7M6LZF53ZUOVMA2AQR7MEQ, seg_pos={0 0}, upload=476427]: Done uploading piece. err: <nil>
Dec 28 06:45:55.256651329 (543h16m55.127734374s) [noise=true , address=77.90.31.18:4805 , piece_id=JHRH2QGRYW2PXJVJS56NJSMJYUIMZR2GWDS5MB6Q47ENC5IFJB4A, seg_pos={0 0}, upload=476427]: Done uploading piece. err: <nil>
Dec 28 06:45:55.965939406 (543h16m55.837022451s) [noise=true , address=45.91.116.7:4012 , piece_id=UA6DQQODGIBYTNFLAMINYZUZHQLGU27CFTLIVLOKNKWCBCSABPYQ, seg_pos={0 0}, upload=476427]: Done uploading piece. err: <nil>
Dec 28 06:45:56.450601385 (543h16m56.32168442s) [noise=true , address=145.79.227.15:4875 , piece_id=4LPNBHB52QJYWSCFX62ENK4JVXO7JHM6ABVTWCWJALYR4AUKLM3A, seg_pos={0 0}, upload=476427]: Done uploading piece. err: <nil>
Dec 28 06:45:56.489764058 (543h16m56.360847093s) [noise=true , address=145.79.227.6:4258 , piece_id=QTU5PW4EM4TKEUWA3UNI36PLBBVBJS7P3KGIND6VW43CHVR2GSBA, seg_pos={0 0}, upload=476427]: Done uploading piece. err: <nil>
Dec 28 06:45:56.527906668 (543h16m56.398989703s) [noise=true , address=145.79.227.16:4911 , piece_id=O7LIGIMP42EAVQRWKO4VYUFE33EKRTDKUYDOYXCQX7C33ZYTMKPQ, seg_pos={0 0}, upload=476427]: Done uploading piece. err: <nil>
Dec 28 06:45:56.625175194 (543h16m56.496258229s) [noise=true , address=145.79.227.29:4941 , piece_id=QHADVHLW2ZT65NKXOZDSY5VCA7LHVMTTKUMVXUZY7UDOG4DTXBAQ, seg_pos={0 0}, upload=476427]: Done uploading piece. err: <nil>
Dec 28 06:45:56.651168249 (543h16m56.522251284s) [noise=true , address=145.79.227.2:4115 , piece_id=I343NB67NV6ZWPK5OKL4GYOTWD7T576HLH7IMHCP6OCHEPFTD4DQ, seg_pos={0 0}, upload=476427]: Done uploading piece. err: <nil>
Dec 28 06:45:56.677355684 (543h16m56.548438729s) [noise=true , address=145.79.227.42:4123 , piece_id=WNF2JBCTXYTHP3LOCMSPFPZVDLDTOJ7MAM4TPOAMKKS2WKYVLD7A, seg_pos={0 0}, upload=476427]: Done uploading piece. err: <nil>
Dec 28 06:45:56.777217429 (543h16m56.648300464s) [noise=true , address=145.79.227.51:4409 , piece_id=JFKX6FZG5MH6JMZKC5SHWVZJAJUAYHXIULSXLBCP6RNIWM66KJ2Q, seg_pos={0 0}, upload=476427]: Done uploading piece. err: <nil>
Dec 28 06:45:56.883035734 (543h16m56.754118779s) [noise=true , address=45.91.116.10:4083 , piece_id=UQHLCLK6KF7JDKZN7MDAAUQWV6I3UUSZAFU73Z2LLAAMERLLNQRA, seg_pos={0 0}, upload=476427]: Done uploading piece. err: <nil>
Dec 28 06:46:02.661605401 (543h17m2.532688446s) [noise=true , address=145.79.227.39:4015 , piece_id=EKRM55RLEPDPWPCLDHOLQZMUAF7ZEPBLSTR2JV3OOIK5C3WRGJCQ, seg_pos={0 0}, upload=476427]: Done uploading piece. err: <nil>
Dec 28 06:46:02.661631099 (543h17m2.532714144s) [seg_pos={0 0}, upload=476427]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:46:02.662005702 (543h17m2.533088758s) [noise=true , address=145.79.227.19:4079 , piece_id=NCHK5HX2KDH6FJTPMEIWX6T6WJLTIPKLTRSJQJYA7DHKZ63ATANQ, seg_pos={0 0}, upload=476427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud): protocol: write tcp 188.166.225.170:59088->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:59088->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59088->145.79.227.19:4079: use of closed network connection
Dec 28 06:46:02.662019359 (543h17m2.533102404s) [noise=true , address=145.79.227.19:4079 , piece_id=NCHK5HX2KDH6FJTPMEIWX6T6WJLTIPKLTRSJQJYA7DHKZ63ATANQ, seg_pos={0 0}, upload=476427]: Long tail cancellation due to optimal threshold
Dec 28 06:46:02.662105453 (543h17m2.533188498s) [noise=true , address=145.79.227.47:4267 , piece_id=M6QMLGZDNBJKRCYJDT5CG32BMESHXZLHMJY5EQ6JGFNI43NFO2DQ, seg_pos={0 0}, upload=476427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8): protocol: read tcp 188.166.225.170:42474->145.79.227.47:4267: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 06:46:02.662212827 (543h17m2.533295872s) [noise=true , address=145.79.227.47:4267 , piece_id=M6QMLGZDNBJKRCYJDT5CG32BMESHXZLHMJY5EQ6JGFNI43NFO2DQ, seg_pos={0 0}, upload=476427]: Long tail cancellation due to optimal threshold
Dec 28 06:46:02.662621215 (543h17m2.53370426s) [noise=true , address=145.79.227.4:4187 , piece_id=4DPDKL647IAL47ARF2JZ2KRIULRQHD52CDWIF2W24YE2F2KRRRVQ, seg_pos={0 0}, upload=476427]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: context canceled; context canceled
Dec 28 06:46:02.662632877 (543h17m2.533715922s) [noise=true , address=145.79.227.4:4187 , piece_id=4DPDKL647IAL47ARF2JZ2KRIULRQHD52CDWIF2W24YE2F2KRRRVQ, seg_pos={0 0}, upload=476427]: Long tail cancellation due to optimal threshold
Dec 28 06:46:02.662689535 (543h17m2.53377258s) [seg_pos={0 0}, upload=476427]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:46:02.662852705 (543h17m2.53393576s) [upload=476427]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:46:02.662860290 (543h17m2.533943345s) [upload=476427]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:46:02.952262101 (543h17m2.823345146s) [upload=476427]: Done uploading segments...
Dec 28 06:46:02.952285294 (543h17m2.823368329s) [upload=476427]: Upload finished. err: <nil>
Duration: 30.003502041s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [X...X..XX.X...XX.....X................X.X.....X.X.........X.] duration:
5.000638203s trace:
https://jaeger.dev.storj.io/trace/39dabd49768c3cc2
FAIL (2025-12-28 06:46:17 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00067361s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [.X...X..XX.X...XX.....X................X.X.....X.X.........X] duration:
5.001479203s trace:
https://jaeger.dev.storj.io/trace/39dabd49768c3cc2
FAIL (2025-12-28 06:45:51 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001505985s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [X...X..X.XX.XXX..X..X.........X.XX..X....X.XX.........X.XXX.] duration:
9.821787137s trace:
https://jaeger.dev.storj.io/trace/660ac8667d0ac411
FAIL (2025-12-28 06:45:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: JT4LNQK3YD54CNSHARJ2GL2HH4TAQP63STJHVAPUAU2MF7Z2FINQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: J75I6HSN3QJNOJG4DHQY3YZU4BOVAZTXDTRQL2RAF4YLDMBZZBOA) context deadline exceeded
======LOGS======
Dec 28 06:44:22.266379927 (543h15m22.137462962s) [upload=476404]: Starting upload
Dec 28 06:44:22.266465170 (543h15m22.137548215s) [upload=476404]: Done starting upload
Dec 28 06:44:22.266566903 (543h15m22.137649948s) [upload=476404]: Uploading segments...
Dec 28 06:44:22.266702651 (543h15m22.137785696s) [upload=476404]: Got next segment. Inline: false
Dec 28 06:44:22.266745012 (543h15m22.137828047s) [upload=476404]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:44:22.266751024 (543h15m22.137834059s) [upload=476404]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:44:22.513836695 (543h15m22.384919741s) [seg_pos={0 0}, upload=476404]: Begin upload segment...
Dec 28 06:44:22.513892833 (543h15m22.384975878s) [seg_pos={0 0}, upload=476404]: Done begin upload segment.
Dec 28 06:44:22.513933189 (543h15m22.385016225s) [noise=true , address=77.90.31.10:4519 , piece_id=JT4LNQK3YD54CNSHARJ2GL2HH4TAQP63STJHVAPUAU2MF7Z2FINQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.514085980 (543h15m22.385169015s) [noise=true , address=145.79.227.51:4410 , piece_id=CTXAYTJNJCU2OEJKH6NPG2THB66W4OY6ZPMSAY2PUDRRMFV4M3CQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.514779751 (543h15m22.385862796s) [noise=true , address=77.90.31.15:4697 , piece_id=KZ6PJOERMLFJDIR4VAMDEYS2C2L6VNYJ2L6UDVMK4LZRJZTLBY3Q, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.515214960 (543h15m22.386298005s) [noise=true , address=77.90.31.17:4773 , piece_id=LFFWIMYODUGDOEOAH6RDIFIMKTD5E4XLEW53TTU6RWPDTLHOO2VQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.515655550 (543h15m22.386738595s) [noise=true , address=45.91.116.11:4118 , piece_id=NPE5VQPGXH5GG7GS4I5R65BDUSLBVHE5U5O72KLUVTDDQ74OZANQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.516070440 (543h15m22.387153475s) [noise=true , address=45.91.116.5:4300 , piece_id=J75I6HSN3QJNOJG4DHQY3YZU4BOVAZTXDTRQL2RAF4YLDMBZZBOA, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.516476863 (543h15m22.387559908s) [noise=true , address=145.79.227.38:4977 , piece_id=5MSEMPYIGYVNWALYR3HOPCETKJKM7QMY6W3GPZDMBBOGZHPDAUTQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.516880462 (543h15m22.387963497s) [noise=true , address=145.79.227.19:4079 , piece_id=JJECU3G4FQCRGBVE7HB7FZQGHF4X7ICUL2C2M7UW6FVKXJVRLZZQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.518901160 (543h15m22.389984205s) [noise=true , address=45.91.116.12:4226 , piece_id=EP6JZTVV3KTWWEQPAJKRNLW3NYJCHQQXPM2SVIVK7ZTTBW3CIKEQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.519864524 (543h15m22.390947569s) [noise=true , address=145.79.227.3:4151 , piece_id=GLXRVX66UJQBMX2T6F6VVT7LLP7G4Y7CLCJ2ZRQQOTPKDCDSZ7BQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.521959924 (543h15m22.393042969s) [noise=true , address=145.79.227.10:4401 , piece_id=2QTMVAQTW7Z7JIKZM3B4SYMA3YYXZOZVRGLCLBLOITCVDGR3RBJQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.523739843 (543h15m22.394822888s) [noise=true , address=145.79.227.11:4732 , piece_id=TIHKILYHDQVYXC3HKNT5ZW7HFTF2O24FWTQIOUVAHW6ZYRAGVL6A, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.525279996 (543h15m22.396363041s) [noise=true , address=145.79.227.17:4972 , piece_id=B6BCJOOQXRLZYIOLZ3H5O3CAAWBGBIGGNFVQTWQO4P4R7MBPNBSA, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.527010020 (543h15m22.398093065s) [noise=true , address=145.79.227.50:4375 , piece_id=A2NZJ7IXZZDICVIT2Y5HDQBGDNHV5ODN6GKWHPDXM4Z4V7JJMYHQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.528981363 (543h15m22.400064409s) [noise=true , address=145.79.227.18:4006 , piece_id=3ISLEJRPTFM6TI6AIRFNXH2PXYFDZER3R432ZGIN75MPPCPXHNPQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.529593490 (543h15m22.400676535s) [noise=true , address=145.79.227.48:4302 , piece_id=DIX2YVM6MVGPWW6DXL4DKPIXB2LKOPLQ7YFA32CSTHUKPVM4N6PA, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:24.233559474 (543h15m24.10464252s) [upload=476404]: Next returned nil segment
Dec 28 06:44:24.233589311 (543h15m24.104672346s) [upload=476404]: Waiting for error group managing segments...
Dec 28 06:44:24.509277849 (543h15m24.380360884s) [noise=true , address=145.79.227.38:4977 , piece_id=5MSEMPYIGYVNWALYR3HOPCETKJKM7QMY6W3GPZDMBBOGZHPDAUTQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.526028976 (543h15m24.397112021s) [noise=true , address=145.79.227.19:4079 , piece_id=JJECU3G4FQCRGBVE7HB7FZQGHF4X7ICUL2C2M7UW6FVKXJVRLZZQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.530915872 (543h15m24.401998917s) [noise=true , address=145.79.227.48:4302 , piece_id=DIX2YVM6MVGPWW6DXL4DKPIXB2LKOPLQ7YFA32CSTHUKPVM4N6PA, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.533782721 (543h15m24.404865766s) [noise=true , address=145.79.227.10:4401 , piece_id=2QTMVAQTW7Z7JIKZM3B4SYMA3YYXZOZVRGLCLBLOITCVDGR3RBJQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.536117897 (543h15m24.407200932s) [noise=true , address=145.79.227.18:4006 , piece_id=3ISLEJRPTFM6TI6AIRFNXH2PXYFDZER3R432ZGIN75MPPCPXHNPQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.536381299 (543h15m24.407464334s) [noise=true , address=145.79.227.50:4375 , piece_id=A2NZJ7IXZZDICVIT2Y5HDQBGDNHV5ODN6GKWHPDXM4Z4V7JJMYHQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.540322223 (543h15m24.411405258s) [noise=true , address=145.79.227.51:4410 , piece_id=CTXAYTJNJCU2OEJKH6NPG2THB66W4OY6ZPMSAY2PUDRRMFV4M3CQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.555597641 (543h15m24.426680686s) [noise=true , address=145.79.227.11:4732 , piece_id=TIHKILYHDQVYXC3HKNT5ZW7HFTF2O24FWTQIOUVAHW6ZYRAGVL6A, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.556029272 (543h15m24.427112317s) [noise=true , address=145.79.227.17:4972 , piece_id=B6BCJOOQXRLZYIOLZ3H5O3CAAWBGBIGGNFVQTWQO4P4R7MBPNBSA, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.564501234 (543h15m24.435584269s) [noise=true , address=145.79.227.3:4151 , piece_id=GLXRVX66UJQBMX2T6F6VVT7LLP7G4Y7CLCJ2ZRQQOTPKDCDSZ7BQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.809970048 (543h15m24.681053093s) [noise=true , address=45.91.116.5:4300 , piece_id=J75I6HSN3QJNOJG4DHQY3YZU4BOVAZTXDTRQL2RAF4YLDMBZZBOA, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.811090032 (543h15m24.682173077s) [noise=true , address=77.90.31.15:4697 , piece_id=KZ6PJOERMLFJDIR4VAMDEYS2C2L6VNYJ2L6UDVMK4LZRJZTLBY3Q, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.834962418 (543h15m24.706045453s) [noise=true , address=77.90.31.10:4519 , piece_id=JT4LNQK3YD54CNSHARJ2GL2HH4TAQP63STJHVAPUAU2MF7Z2FINQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.834983268 (543h15m24.706066303s) [seg_pos={0 0}, upload=476404]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:44:24.835237632 (543h15m24.706320667s) [noise=true , address=45.91.116.11:4118 , piece_id=NPE5VQPGXH5GG7GS4I5R65BDUSLBVHE5U5O72KLUVTDDQ74OZANQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF
Dec 28 06:44:24.835246089 (543h15m24.706329134s) [noise=true , address=45.91.116.11:4118 , piece_id=NPE5VQPGXH5GG7GS4I5R65BDUSLBVHE5U5O72KLUVTDDQ74OZANQ, seg_pos={0 0}, upload=476404]: Long tail cancellation due to optimal threshold
Dec 28 06:44:24.835401264 (543h15m24.706484309s) [noise=true , address=77.90.31.17:4773 , piece_id=LFFWIMYODUGDOEOAH6RDIFIMKTD5E4XLEW53TTU6RWPDTLHOO2VQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 28 06:44:24.835424719 (543h15m24.706507764s) [noise=true , address=77.90.31.17:4773 , piece_id=LFFWIMYODUGDOEOAH6RDIFIMKTD5E4XLEW53TTU6RWPDTLHOO2VQ, seg_pos={0 0}, upload=476404]: Long tail cancellation due to optimal threshold
Dec 28 06:44:24.835610883 (543h15m24.706693928s) [noise=true , address=45.91.116.12:4226 , piece_id=EP6JZTVV3KTWWEQPAJKRNLW3NYJCHQQXPM2SVIVK7ZTTBW3CIKEQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF
Dec 28 06:44:24.835620672 (543h15m24.706703707s) [noise=true , address=45.91.116.12:4226 , piece_id=EP6JZTVV3KTWWEQPAJKRNLW3NYJCHQQXPM2SVIVK7ZTTBW3CIKEQ, seg_pos={0 0}, upload=476404]: Long tail cancellation due to optimal threshold
Dec 28 06:44:24.835665688 (543h15m24.706748733s) [seg_pos={0 0}, upload=476404]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:44:24.835779344 (543h15m24.706862399s) [upload=476404]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:44:24.835785917 (543h15m24.706868962s) [upload=476404]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:44:25.134403596 (543h15m25.005486641s) [upload=476404]: Done uploading segments...
Dec 28 06:44:25.134432170 (543h15m25.005515206s) [upload=476404]: Upload finished. err: <nil>
Duration: 9.821811865s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.X...X..X.XX.XXX..X..X.........X.XX..X....X.XX.........X.XXX] duration:
30.002920827s trace:
https://jaeger.dev.storj.io/trace/660ac8667d0ac411
FAIL (2025-12-28 06:44:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: JT4LNQK3YD54CNSHARJ2GL2HH4TAQP63STJHVAPUAU2MF7Z2FINQ) context deadline exceeded; piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: J75I6HSN3QJNOJG4DHQY3YZU4BOVAZTXDTRQL2RAF4YLDMBZZBOA) context deadline exceeded
======LOGS======
Dec 28 06:44:22.266379927 (543h15m22.137462962s) [upload=476404]: Starting upload
Dec 28 06:44:22.266465170 (543h15m22.137548215s) [upload=476404]: Done starting upload
Dec 28 06:44:22.266566903 (543h15m22.137649948s) [upload=476404]: Uploading segments...
Dec 28 06:44:22.266702651 (543h15m22.137785696s) [upload=476404]: Got next segment. Inline: false
Dec 28 06:44:22.266745012 (543h15m22.137828047s) [upload=476404]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:44:22.266751024 (543h15m22.137834059s) [upload=476404]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:44:22.513836695 (543h15m22.384919741s) [seg_pos={0 0}, upload=476404]: Begin upload segment...
Dec 28 06:44:22.513892833 (543h15m22.384975878s) [seg_pos={0 0}, upload=476404]: Done begin upload segment.
Dec 28 06:44:22.513933189 (543h15m22.385016225s) [noise=true , address=77.90.31.10:4519 , piece_id=JT4LNQK3YD54CNSHARJ2GL2HH4TAQP63STJHVAPUAU2MF7Z2FINQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.514085980 (543h15m22.385169015s) [noise=true , address=145.79.227.51:4410 , piece_id=CTXAYTJNJCU2OEJKH6NPG2THB66W4OY6ZPMSAY2PUDRRMFV4M3CQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.514779751 (543h15m22.385862796s) [noise=true , address=77.90.31.15:4697 , piece_id=KZ6PJOERMLFJDIR4VAMDEYS2C2L6VNYJ2L6UDVMK4LZRJZTLBY3Q, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.515214960 (543h15m22.386298005s) [noise=true , address=77.90.31.17:4773 , piece_id=LFFWIMYODUGDOEOAH6RDIFIMKTD5E4XLEW53TTU6RWPDTLHOO2VQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.515655550 (543h15m22.386738595s) [noise=true , address=45.91.116.11:4118 , piece_id=NPE5VQPGXH5GG7GS4I5R65BDUSLBVHE5U5O72KLUVTDDQ74OZANQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.516070440 (543h15m22.387153475s) [noise=true , address=45.91.116.5:4300 , piece_id=J75I6HSN3QJNOJG4DHQY3YZU4BOVAZTXDTRQL2RAF4YLDMBZZBOA, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.516476863 (543h15m22.387559908s) [noise=true , address=145.79.227.38:4977 , piece_id=5MSEMPYIGYVNWALYR3HOPCETKJKM7QMY6W3GPZDMBBOGZHPDAUTQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.516880462 (543h15m22.387963497s) [noise=true , address=145.79.227.19:4079 , piece_id=JJECU3G4FQCRGBVE7HB7FZQGHF4X7ICUL2C2M7UW6FVKXJVRLZZQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.518901160 (543h15m22.389984205s) [noise=true , address=45.91.116.12:4226 , piece_id=EP6JZTVV3KTWWEQPAJKRNLW3NYJCHQQXPM2SVIVK7ZTTBW3CIKEQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.519864524 (543h15m22.390947569s) [noise=true , address=145.79.227.3:4151 , piece_id=GLXRVX66UJQBMX2T6F6VVT7LLP7G4Y7CLCJ2ZRQQOTPKDCDSZ7BQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.521959924 (543h15m22.393042969s) [noise=true , address=145.79.227.10:4401 , piece_id=2QTMVAQTW7Z7JIKZM3B4SYMA3YYXZOZVRGLCLBLOITCVDGR3RBJQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.523739843 (543h15m22.394822888s) [noise=true , address=145.79.227.11:4732 , piece_id=TIHKILYHDQVYXC3HKNT5ZW7HFTF2O24FWTQIOUVAHW6ZYRAGVL6A, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.525279996 (543h15m22.396363041s) [noise=true , address=145.79.227.17:4972 , piece_id=B6BCJOOQXRLZYIOLZ3H5O3CAAWBGBIGGNFVQTWQO4P4R7MBPNBSA, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.527010020 (543h15m22.398093065s) [noise=true , address=145.79.227.50:4375 , piece_id=A2NZJ7IXZZDICVIT2Y5HDQBGDNHV5ODN6GKWHPDXM4Z4V7JJMYHQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.528981363 (543h15m22.400064409s) [noise=true , address=145.79.227.18:4006 , piece_id=3ISLEJRPTFM6TI6AIRFNXH2PXYFDZER3R432ZGIN75MPPCPXHNPQ, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:22.529593490 (543h15m22.400676535s) [noise=true , address=145.79.227.48:4302 , piece_id=DIX2YVM6MVGPWW6DXL4DKPIXB2LKOPLQ7YFA32CSTHUKPVM4N6PA, seg_pos={0 0}, upload=476404]: Uploading piece...
Dec 28 06:44:24.233559474 (543h15m24.10464252s) [upload=476404]: Next returned nil segment
Dec 28 06:44:24.233589311 (543h15m24.104672346s) [upload=476404]: Waiting for error group managing segments...
Dec 28 06:44:24.509277849 (543h15m24.380360884s) [noise=true , address=145.79.227.38:4977 , piece_id=5MSEMPYIGYVNWALYR3HOPCETKJKM7QMY6W3GPZDMBBOGZHPDAUTQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.526028976 (543h15m24.397112021s) [noise=true , address=145.79.227.19:4079 , piece_id=JJECU3G4FQCRGBVE7HB7FZQGHF4X7ICUL2C2M7UW6FVKXJVRLZZQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.530915872 (543h15m24.401998917s) [noise=true , address=145.79.227.48:4302 , piece_id=DIX2YVM6MVGPWW6DXL4DKPIXB2LKOPLQ7YFA32CSTHUKPVM4N6PA, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.533782721 (543h15m24.404865766s) [noise=true , address=145.79.227.10:4401 , piece_id=2QTMVAQTW7Z7JIKZM3B4SYMA3YYXZOZVRGLCLBLOITCVDGR3RBJQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.536117897 (543h15m24.407200932s) [noise=true , address=145.79.227.18:4006 , piece_id=3ISLEJRPTFM6TI6AIRFNXH2PXYFDZER3R432ZGIN75MPPCPXHNPQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.536381299 (543h15m24.407464334s) [noise=true , address=145.79.227.50:4375 , piece_id=A2NZJ7IXZZDICVIT2Y5HDQBGDNHV5ODN6GKWHPDXM4Z4V7JJMYHQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.540322223 (543h15m24.411405258s) [noise=true , address=145.79.227.51:4410 , piece_id=CTXAYTJNJCU2OEJKH6NPG2THB66W4OY6ZPMSAY2PUDRRMFV4M3CQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.555597641 (543h15m24.426680686s) [noise=true , address=145.79.227.11:4732 , piece_id=TIHKILYHDQVYXC3HKNT5ZW7HFTF2O24FWTQIOUVAHW6ZYRAGVL6A, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.556029272 (543h15m24.427112317s) [noise=true , address=145.79.227.17:4972 , piece_id=B6BCJOOQXRLZYIOLZ3H5O3CAAWBGBIGGNFVQTWQO4P4R7MBPNBSA, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.564501234 (543h15m24.435584269s) [noise=true , address=145.79.227.3:4151 , piece_id=GLXRVX66UJQBMX2T6F6VVT7LLP7G4Y7CLCJ2ZRQQOTPKDCDSZ7BQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.809970048 (543h15m24.681053093s) [noise=true , address=45.91.116.5:4300 , piece_id=J75I6HSN3QJNOJG4DHQY3YZU4BOVAZTXDTRQL2RAF4YLDMBZZBOA, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.811090032 (543h15m24.682173077s) [noise=true , address=77.90.31.15:4697 , piece_id=KZ6PJOERMLFJDIR4VAMDEYS2C2L6VNYJ2L6UDVMK4LZRJZTLBY3Q, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.834962418 (543h15m24.706045453s) [noise=true , address=77.90.31.10:4519 , piece_id=JT4LNQK3YD54CNSHARJ2GL2HH4TAQP63STJHVAPUAU2MF7Z2FINQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: <nil>
Dec 28 06:44:24.834983268 (543h15m24.706066303s) [seg_pos={0 0}, upload=476404]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:44:24.835237632 (543h15m24.706320667s) [noise=true , address=45.91.116.11:4118 , piece_id=NPE5VQPGXH5GG7GS4I5R65BDUSLBVHE5U5O72KLUVTDDQ74OZANQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF
Dec 28 06:44:24.835246089 (543h15m24.706329134s) [noise=true , address=45.91.116.11:4118 , piece_id=NPE5VQPGXH5GG7GS4I5R65BDUSLBVHE5U5O72KLUVTDDQ74OZANQ, seg_pos={0 0}, upload=476404]: Long tail cancellation due to optimal threshold
Dec 28 06:44:24.835401264 (543h15m24.706484309s) [noise=true , address=77.90.31.17:4773 , piece_id=LFFWIMYODUGDOEOAH6RDIFIMKTD5E4XLEW53TTU6RWPDTLHOO2VQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 28 06:44:24.835424719 (543h15m24.706507764s) [noise=true , address=77.90.31.17:4773 , piece_id=LFFWIMYODUGDOEOAH6RDIFIMKTD5E4XLEW53TTU6RWPDTLHOO2VQ, seg_pos={0 0}, upload=476404]: Long tail cancellation due to optimal threshold
Dec 28 06:44:24.835610883 (543h15m24.706693928s) [noise=true , address=45.91.116.12:4226 , piece_id=EP6JZTVV3KTWWEQPAJKRNLW3NYJCHQQXPM2SVIVK7ZTTBW3CIKEQ, seg_pos={0 0}, upload=476404]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF
Dec 28 06:44:24.835620672 (543h15m24.706703707s) [noise=true , address=45.91.116.12:4226 , piece_id=EP6JZTVV3KTWWEQPAJKRNLW3NYJCHQQXPM2SVIVK7ZTTBW3CIKEQ, seg_pos={0 0}, upload=476404]: Long tail cancellation due to optimal threshold
Dec 28 06:44:24.835665688 (543h15m24.706748733s) [seg_pos={0 0}, upload=476404]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:44:24.835779344 (543h15m24.706862399s) [upload=476404]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:44:24.835785917 (543h15m24.706868962s) [upload=476404]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:44:25.134403596 (543h15m25.005486641s) [upload=476404]: Done uploading segments...
Dec 28 06:44:25.134432170 (543h15m25.005515206s) [upload=476404]: Upload finished. err: <nil>
Duration: 30.002946986s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
12 out of 60 is failed (threshold=6) [..X...X..XX.X...XX.....X................X.X.....X.X.........] duration:
3.616993847s trace:
https://jaeger.dev.storj.io/trace/6af3578081c62393
FAIL (2025-12-28 06:44:38 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.617024486s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [..X...X..X.XX.XXX..X..X.........X.XX..X....X.XX.........X.XX] duration:
30.003882446s trace:
https://jaeger.dev.storj.io/trace/5a36a198e1cb9dc3
FAIL (2025-12-28 06:43:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: KPVTZ6XJ3I3OLS4ZD54VCCKMYUXRCWWAHP5NAKAAHVPFPLYE2WIQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: ELPZ6MLFOOMN4WLMUVY5KIZSEJWY2GA3TY2UGTQM5A3XKF5C2OEQ) context deadline exceeded
======LOGS======
28 06:43:28.750494505 (543h14m28.62157755s) [upload=476392]: Starting upload
Dec 28 06:43:28.750577243 (543h14m28.621660308s) [upload=476392]: Done starting upload
Dec 28 06:43:28.750643689 (543h14m28.621726734s) [upload=476392]: Uploading segments...
Dec 28 06:43:28.750919314 (543h14m28.622002359s) [upload=476392]: Got next segment. Inline: false
Dec 28 06:43:28.750966033 (543h14m28.622049078s) [upload=476392]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:43:28.750972676 (543h14m28.622055711s) [upload=476392]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:43:28.999039207 (543h14m28.870122242s) [seg_pos={0 0}, upload=476392]: Begin upload segment...
Dec 28 06:43:28.999086366 (543h14m28.870169401s) [seg_pos={0 0}, upload=476392]: Done begin upload segment.
Dec 28 06:43:28.999111414 (543h14m28.870194459s) [noise=true , address=145.79.227.14:4837 , piece_id=2DEXNKAPURU47UXIIHXYFUMIN5G2GXJNFP6D2LIDRMAU3ELQIPPA, seg_pos={0 0}, upload=476392]: Uploading piece...
Dec 28 06:43:28.999226934 (543h14m28.870309969s) [noise=true , address=145.79.227.39:4015 , piece_id=OEEGLVE2H5JEYD4L4L5TGCHB6FTRFM5ORYDRZTG7WJU3JFENK3DQ, seg_pos={0 0}, upload=476392]: Uploading piece...
Dec 28 06:43:28.999864779 (543h14m28.870947834s) [noise=true , address=45.91.116.4:4940 , piece_id=SJXYEZEVYGCAPY5J72KTUZ5G5CM5T4FFDSUZOSMJMY637GCPV2GA, seg_pos={0 0}, upload=476392]: Uploading piece...
Dec 28 06:43:29.000323703 (543h14m28.871406748s) [noise=true , address=77.90.31.14:4664 , piece_id=TCFTHB72RBBRHJDZA33FVGS7C45ENA6YZS7U4MB37PPFOHQDHFLA, seg_pos={0 0}, upload=476392]: Uploading piece...
Dec 28 06:43:29.000781103 (543h14m28.871864148s) [noise=true , address=145.79.227.37:4566 , piece_id=3RDXNUHWDLMPTODWABUZ54RG2GAS2PC2JFQF3C6445MTJ3JQWVVA, seg_pos={0 0}, upload=476392]: Uploading piece...
Dec 28 06:43:29.001299400 (543h14m28.872382445s) [noise=true , address=45.91.116.12:4228 , piece_id=OOWOU7HHX56CQLEH4CUKQL5RK5TPVUTQH774XGWX7NSD3QNLNHEQ, seg_pos={0 0}, upload=476392]: Uploading piece...
Dec 28 06:43:29.001882750 (543h14m28.872965796s) [noise=true , address=145.79.227.36:4519 , piece_id=K62OD7A3CRZGH27TESXVVT4RVDJMBOQVTVB2ENMU6TM4QEUMPUCQ, seg_pos={0 0}, upload=476392]: Uploading piece...
Dec 28 06:43:29.002895079 (543h14m28.873978114s) [noise=true , address=145.79.227.38:4979 , piece_id=5UCT6MSKCS2DR7BTL4U5BJCVI47AW2NWGRLYZLPYJPZSYQTUNAFQ, seg_pos={0 0}, upload=476392]: Uploading piece...
Dec 28 06:43:29.003844747 (543h14m28.874927782s) [noise=true , address=77.90.31.18:4805 , piece_id=KTY27XXRPAUNP5RDSU7OWF3HRI2KBAEHHHB2JIFMJJRV6TI2VLZA, seg_pos={0 0}, upload=476392]: Uploading piece...
Dec 28 06:43:29.004268784 (543h14m28.875351819s) [noise=true , address=145.79.227.18:4007 , piece_id=P7WOF5QLXRMTHS45YOFI4YIYSBIMN6CCJLBMAZMUT2ADRQDZEXBA, seg_pos={0 0}, upload=476392]: Uploading piece...
Dec 28 06:43:29.005528793 (543h14m28.876611828s) [noise=true , address=145.79.227.3:4151 , piece_id=Z3YZJRP3ATGH24DRIGTIINO6JG4TDGKJS5C4IO6X4ITDVO2MKWUA, seg_pos={0 0}, upload=476392]: Uploading piece...
Dec 28 06:43:29.005961578 (543h14m28.877044643s) [noise=true , address=145.79.227.46:4603 , piece_id=HX5SRS6IESYNQ4DOGML4JU7JDTDOJOH6PJILRNQFDUTX27IXKBAA, seg_pos={0 0}, upload=476392]: Uploading piece...
Dec 28 06:43:29.007217409 (543h14m28.878300464s) [noise=true , address=45.91.116.13:4156 , piece_id=KPVTZ6XJ3I3OLS4ZD54VCCKMYUXRCWWAHP5NAKAAHVPFPLYE2WIQ, seg_pos={0 0}, upload=476392]: Uploading piece...
Dec 28 06:43:29.008478320 (543h14m28.879561365s) [noise=true , address=145.79.227.48:4303 , piece_id=SFMBT4AQL6WJGIYVCUJJYD7PXJRA4S5OZPNEULX75JFWCYDUWKHQ, seg_pos={0 0}, upload=476392]: Uploading piece...
Dec 28 06:43:29.011510374 (543h14m28.882593409s) [noise=true , address=145.79.227.16:4911 , piece_id=IMDKBONWJ6R4G6W5XBV2AHBJFP2YQLO6OUNDQOXVNBZ4X7DHYXDQ, seg_pos={0 0}, upload=476392]: Uploading piece...
Dec 28 06:43:29.015042530 (543h14m28.886125575s) [noise=true , address=77.90.31.13:4629 , piece_id=ELPZ6MLFOOMN4WLMUVY5KIZSEJWY2GA3TY2UGTQM5A3XKF5C2OEQ, seg_pos={0 0}, upload=476392]: Uploading piece...
Dec 28 06:43:30.734413196 (543h14m30.605496241s) [upload=476392]: Next returned nil segment
Dec 28 06:43:30.734458782 (543h14m30.605541827s) [upload=476392]: Waiting for error group managing segments...
Dec 28 06:43:31.019086352 (543h14m30.890169397s) [noise=true , address=145.79.227.46:4603 , piece_id=HX5SRS6IESYNQ4DOGML4JU7JDTDOJOH6PJILRNQFDUTX27IXKBAA, seg_pos={0 0}, upload=476392]: Done uploading piece. err: <nil>
Dec 28 06:43:31.020122615 (543h14m30.89120566s) [noise=true , address=145.79.227.3:4151 , piece_id=Z3YZJRP3ATGH24DRIGTIINO6JG4TDGKJS5C4IO6X4ITDVO2MKWUA, seg_pos={0 0}, upload=476392]: Done uploading piece. err: <nil>
Dec 28 06:43:31.028549742 (543h14m30.899632777s) [noise=true , address=145.79.227.18:4007 , piece_id=P7WOF5QLXRMTHS45YOFI4YIYSBIMN6CCJLBMAZMUT2ADRQDZEXBA, seg_pos={0 0}, upload=476392]: Done uploading piece. err: <nil>
Dec 28 06:43:31.029102494 (543h14m30.900185539s) [noise=true , address=145.79.227.48:4303 , piece_id=SFMBT4AQL6WJGIYVCUJJYD7PXJRA4S5OZPNEULX75JFWCYDUWKHQ, seg_pos={0 0}, upload=476392]: Done uploading piece. err: <nil>
Dec 28 06:43:31.032456911 (543h14m30.903539946s) [noise=true , address=145.79.227.39:4015 , piece_id=OEEGLVE2H5JEYD4L4L5TGCHB6FTRFM5ORYDRZTG7WJU3JFENK3DQ, seg_pos={0 0}, upload=476392]: Done uploading piece. err: <nil>
Dec 28 06:43:31.033182012 (543h14m30.904265057s) [noise=true , address=145.79.227.36:4519 , piece_id=K62OD7A3CRZGH27TESXVVT4RVDJMBOQVTVB2ENMU6TM4QEUMPUCQ, seg_pos={0 0}, upload=476392]: Done uploading piece. err: <nil>
Dec 28 06:43:31.034543575 (543h14m30.90562661s) [noise=true , address=145.79.227.38:4979 , piece_id=5UCT6MSKCS2DR7BTL4U5BJCVI47AW2NWGRLYZLPYJPZSYQTUNAFQ, seg_pos={0 0}, upload=476392]: Done uploading piece. err: <nil>
Dec 28 06:43:31.041213456 (543h14m30.912296501s) [noise=true , address=145.79.227.37:4566 , piece_id=3RDXNUHWDLMPTODWABUZ54RG2GAS2PC2JFQF3C6445MTJ3JQWVVA, seg_pos={0 0}, upload=476392]: Done uploading piece. err: <nil>
Dec 28 06:43:31.052276662 (543h14m30.923359697s) [noise=true , address=145.79.227.14:4837 , piece_id=2DEXNKAPURU47UXIIHXYFUMIN5G2GXJNFP6D2LIDRMAU3ELQIPPA, seg_pos={0 0}, upload=476392]: Done uploading piece. err: <nil>
Dec 28 06:43:31.059158987 (543h14m30.930242032s) [noise=true , address=145.79.227.16:4911 , piece_id=IMDKBONWJ6R4G6W5XBV2AHBJFP2YQLO6OUNDQOXVNBZ4X7DHYXDQ, seg_pos={0 0}, upload=476392]: Done uploading piece. err: <nil>
Dec 28 06:43:31.229470960 (543h14m31.100554005s) [noise=true , address=77.90.31.14:4664 , piece_id=TCFTHB72RBBRHJDZA33FVGS7C45ENA6YZS7U4MB37PPFOHQDHFLA, seg_pos={0 0}, upload=476392]: Done uploading piece. err: <nil>
Dec 28 06:43:31.294355962 (543h14m31.165439007s) [noise=true , address=77.90.31.13:4629 , piece_id=ELPZ6MLFOOMN4WLMUVY5KIZSEJWY2GA3TY2UGTQM5A3XKF5C2OEQ, seg_pos={0 0}, upload=476392]: Done uploading piece. err: <nil>
Dec 28 06:43:31.297158108 (543h14m31.168241143s) [noise=true , address=45.91.116.13:4156 , piece_id=KPVTZ6XJ3I3OLS4ZD54VCCKMYUXRCWWAHP5NAKAAHVPFPLYE2WIQ, seg_pos={0 0}, upload=476392]: Done uploading piece. err: <nil>
Dec 28 06:43:31.297170852 (543h14m31.168253887s) [seg_pos={0 0}, upload=476392]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:43:31.297359301 (543h14m31.168442337s) [noise=true , address=77.90.31.18:4805 , piece_id=KTY27XXRPAUNP5RDSU7OWF3HRI2KBAEHHHB2JIFMJJRV6TI2VLZA, seg_pos={0 0}, upload=476392]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: expected piece hash; context canceled; EOF
Dec 28 06:43:31.297367858 (543h14m31.168450893s) [noise=true , address=77.90.31.18:4805 , piece_id=KTY27XXRPAUNP5RDSU7OWF3HRI2KBAEHHHB2JIFMJJRV6TI2VLZA, seg_pos={0 0}, upload=476392]: Long tail cancellation due to optimal threshold
Dec 28 06:43:31.297678659 (543h14m31.168761704s) [noise=true , address=45.91.116.4:4940 , piece_id=SJXYEZEVYGCAPY5J72KTUZ5G5CM5T4FFDSUZOSMJMY637GCPV2GA, seg_pos={0 0}, upload=476392]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:46558->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:46558->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46558->45.91.116.4:4940: use of closed network connection
Dec 28 06:43:31.297685943 (543h14m31.168768988s) [noise=true , address=45.91.116.4:4940 , piece_id=SJXYEZEVYGCAPY5J72KTUZ5G5CM5T4FFDSUZOSMJMY637GCPV2GA, seg_pos={0 0}, upload=476392]: Long tail cancellation due to optimal threshold
Dec 28 06:43:31.297883229 (543h14m31.168966264s) [noise=true , address=45.91.116.12:4228 , piece_id=OOWOU7HHX56CQLEH4CUKQL5RK5TPVUTQH774XGWX7NSD3QNLNHEQ, seg_pos={0 0}, upload=476392]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF
Dec 28 06:43:31.297891835 (543h14m31.16897487s) [noise=true , address=45.91.116.12:4228 , piece_id=OOWOU7HHX56CQLEH4CUKQL5RK5TPVUTQH774XGWX7NSD3QNLNHEQ, seg_pos={0 0}, upload=476392]: Long tail cancellation due to optimal threshold
Dec 28 06:43:31.297915140 (543h14m31.168998175s) [seg_pos={0 0}, upload=476392]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:43:31.298002005 (543h14m31.16908504s) [upload=476392]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:43:31.298011252 (543h14m31.169094327s) [upload=476392]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:43:31.589009182 (543h14m31.460092217s) [upload=476392]: Done uploading segments...
Dec 28 06:43:31.589029430 (543h14m31.460112465s) [upload=476392]: Upload finished. err: <nil>
Duration: 30.003906392s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
13 out of 60 is failed (threshold=6) [X..X...X..XX.X...XX.....X................X.X.....X.X........] duration:
4.006608269s trace:
https://jaeger.dev.storj.io/trace/6af3578081c62393
FAIL (2025-12-28 06:43:54 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.006636853s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [...X...X..X.XX.XXX..X..X.........X.XX..X....X.XX.........X.X] duration:
30.003596119s trace:
https://jaeger.dev.storj.io/trace/9e423e99b458b6b
FAIL (2025-12-28 06:43:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
egments...
Dec 28 06:42:37.539512514 (543h13m37.410595569s) [upload=476378]: Got next segment. Inline: false
Dec 28 06:42:37.539604960 (543h13m37.410688005s) [upload=476378]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:42:37.539613496 (543h13m37.410696531s) [upload=476378]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:42:37.784344215 (543h13m37.65542726s) [seg_pos={0 0}, upload=476378]: Begin upload segment...
Dec 28 06:42:37.784400082 (543h13m37.655483117s) [seg_pos={0 0}, upload=476378]: Done begin upload segment.
Dec 28 06:42:37.784461188 (543h13m37.655544243s) [noise=true , address=45.91.116.7:4011 , piece_id=QE35VFMSAK56VJFON5AYNICZIPJISO5FK5AAVABJVRX2HP2FQMWA, seg_pos={0 0}, upload=476378]: Uploading piece...
Dec 28 06:42:37.784499140 (543h13m37.655582175s) [noise=true , address=145.79.227.32:4835 , piece_id=ADIOT5NRV5DPYTWSDT66R74FOZ3PZ76LKGUGUOJCENRPFJZHOVEQ, seg_pos={0 0}, upload=476378]: Uploading piece...
Dec 28 06:42:37.785262233 (543h13m37.656345278s) [noise=true , address=145.79.227.23:4547 , piece_id=GRAAYBMZS6ZYF5XJOW2U57PJ64QKQJE56AMUQYZ4KF5QM2ZZKCTQ, seg_pos={0 0}, upload=476378]: Uploading piece...
Dec 28 06:42:37.785359849 (543h13m37.656442894s) [noise=true , address=145.79.227.36:4519 , piece_id=MI5SDSKIIM3U5V2X3DUWWOPKHI5XUVGRNA3GQSMB57J4W4XHKY2A, seg_pos={0 0}, upload=476378]: Uploading piece...
Dec 28 06:42:37.785719835 (543h13m37.65680288s) [noise=true , address=145.79.227.45:4230 , piece_id=IWHNS5W5TVQPX6IJGRB4WHUNTLYC6XZGKHFJ6WDPQ4LLVWTJM6BQ, seg_pos={0 0}, upload=476378]: Uploading piece...
Dec 28 06:42:37.786309909 (543h13m37.657392954s) [noise=true , address=45.91.116.11:4118 , piece_id=UJH4KP52P4BTYVTOOAZ4P6MYZKLUNASNJAOIP7C3NQIZONAVXHCA, seg_pos={0 0}, upload=476378]: Uploading piece...
Dec 28 06:42:37.786667840 (543h13m37.657750875s) [noise=true , address=145.79.227.37:4566 , piece_id=RRAYH76DYIUCZWCUUGUL6N4SE5ZGCSKXYC4PWIQY2PNGUVDKI4DA, seg_pos={0 0}, upload=476378]: Uploading piece...
Dec 28 06:42:37.786896245 (543h13m37.65797929s) [noise=true , address=145.79.227.17:4970 , piece_id=TT2UQVT2JYQUVHX3NCTUA4I2JXEQJ7SXYQ75US3Y5V2M5VMROHFQ, seg_pos={0 0}, upload=476378]: Uploading piece...
Dec 28 06:42:37.787458526 (543h13m37.658541561s) [noise=true , address=145.79.227.35:4482 , piece_id=FGLVIP7XPNOLQWS7BNG6YU2IL25QNDNMMYPHZEW3I32IEHKVAA4A, seg_pos={0 0}, upload=476378]: Uploading piece...
Dec 28 06:42:37.787533970 (543h13m37.658617015s) [noise=true , address=145.79.227.48:4302 , piece_id=3Z2Q5VSHCZQJILWEXAFFZUREROHT3W5LCPJPJQ3ALAPQKTO32AZQ, seg_pos={0 0}, upload=476378]: Uploading piece...
Dec 28 06:42:37.787565109 (543h13m37.658648154s) [noise=true , address=145.79.227.16:4912 , piece_id=MH5HZVGQJ5EYF2MIQQXPBIMBDFC45EBQVL2BJ3HREPDV6YVMWJKQ, seg_pos={0 0}, upload=476378]: Uploading piece...
Dec 28 06:42:37.788122270 (543h13m37.659205305s) [noise=true , address=145.79.227.52:4445 , piece_id=3DV3QW7EKIBGDPTEZ47GNWAZPBFLGPFGEGE4ZIFMHA2Y3GBWWJ3Q, seg_pos={0 0}, upload=476378]: Uploading piece...
Dec 28 06:42:37.788448190 (543h13m37.659531225s) [noise=true , address=145.79.227.39:4014 , piece_id=JYQKQLSZ2F6HZCRGAJGEYDLS7BCA5QHRR47JM32IOXDXC5RUT7WQ, seg_pos={0 0}, upload=476378]: Uploading piece...
Dec 28 06:42:37.788758110 (543h13m37.659841155s) [noise=true , address=77.90.31.15:4700 , piece_id=ETRSZPCANUZ4RQKBF24SDMQGROXLA5AMUZ67ADFAWMZR4J2RNNCQ, seg_pos={0 0}, upload=476378]: Uploading piece...
Dec 28 06:42:37.789625903 (543h13m37.660708938s) [noise=true , address=145.79.227.25:4619 , piece_id=GQLYX7CBNAOQ6CGK7NCWH7PGFNKQU5W2LVHZYDAHXNLQX32XX7IQ, seg_pos={0 0}, upload=476378]: Uploading piece...
Dec 28 06:42:37.790398865 (543h13m37.6614819s) [noise=true , address=77.90.31.14:4661 , piece_id=ZDUE5DASOPITW7Z47QGXZEMX3WGB3KV2UAAJ7RGGTUXSJ27BR3BQ, seg_pos={0 0}, upload=476378]: Uploading piece...
Dec 28 06:42:39.494915055 (543h13m39.36599809s) [upload=476378]: Next returned nil segment
Dec 28 06:42:39.494931768 (543h13m39.366014813s) [upload=476378]: Waiting for error group managing segments...
Dec 28 06:42:39.782893336 (543h13m39.653976371s) [noise=true , address=145.79.227.39:4014 , piece_id=JYQKQLSZ2F6HZCRGAJGEYDLS7BCA5QHRR47JM32IOXDXC5RUT7WQ, seg_pos={0 0}, upload=476378]: Done uploading piece. err: <nil>
Dec 28 06:42:39.789346233 (543h13m39.660429268s) [noise=true , address=145.79.227.23:4547 , piece_id=GRAAYBMZS6ZYF5XJOW2U57PJ64QKQJE56AMUQYZ4KF5QM2ZZKCTQ, seg_pos={0 0}, upload=476378]: Done uploading piece. err: <nil>
Dec 28 06:42:39.789781451 (543h13m39.660864496s) [noise=true , address=145.79.227.32:4835 , piece_id=ADIOT5NRV5DPYTWSDT66R74FOZ3PZ76LKGUGUOJCENRPFJZHOVEQ, seg_pos={0 0}, upload=476378]: Done uploading piece. err: <nil>
Dec 28 06:42:39.795443874 (543h13m39.666526919s) [noise=true , address=145.79.227.45:4230 , piece_id=IWHNS5W5TVQPX6IJGRB4WHUNTLYC6XZGKHFJ6WDPQ4LLVWTJM6BQ, seg_pos={0 0}, upload=476378]: Done uploading piece. err: <nil>
Dec 28 06:42:39.795711614 (543h13m39.666794659s) [noise=true , address=145.79.227.16:4912 , piece_id=MH5HZVGQJ5EYF2MIQQXPBIMBDFC45EBQVL2BJ3HREPDV6YVMWJKQ, seg_pos={0 0}, upload=476378]: Done uploading piece. err: <nil>
Dec 28 06:42:39.796768697 (543h13m39.667851732s) [noise=true , address=145.79.227.36:4519 , piece_id=MI5SDSKIIM3U5V2X3DUWWOPKHI5XUVGRNA3GQSMB57J4W4XHKY2A, seg_pos={0 0}, upload=476378]: Done uploading piece. err: <nil>
Dec 28 06:42:39.798313769 (543h13m39.669396804s) [noise=true , address=145.79.227.17:4970 , piece_id=TT2UQVT2JYQUVHX3NCTUA4I2JXEQJ7SXYQ75US3Y5V2M5VMROHFQ, seg_pos={0 0}, upload=476378]: Done uploading piece. err: <nil>
Dec 28 06:42:39.798943428 (543h13m39.670026473s) [noise=true , address=145.79.227.48:4302 , piece_id=3Z2Q5VSHCZQJILWEXAFFZUREROHT3W5LCPJPJQ3ALAPQKTO32AZQ, seg_pos={0 0}, upload=476378]: Done uploading piece. err: <nil>
Dec 28 06:42:39.799391270 (543h13m39.670474305s) [noise=true , address=145.79.227.37:4566 , piece_id=RRAYH76DYIUCZWCUUGUL6N4SE5ZGCSKXYC4PWIQY2PNGUVDKI4DA, seg_pos={0 0}, upload=476378]: Done uploading piece. err: <nil>
Dec 28 06:42:39.801731517 (543h13m39.672814552s) [noise=true , address=145.79.227.52:4445 , piece_id=3DV3QW7EKIBGDPTEZ47GNWAZPBFLGPFGEGE4ZIFMHA2Y3GBWWJ3Q, seg_pos={0 0}, upload=476378]: Done uploading piece. err: <nil>
Dec 28 06:42:39.802869724 (543h13m39.673952759s) [noise=true , address=145.79.227.25:4619 , piece_id=GQLYX7CBNAOQ6CGK7NCWH7PGFNKQU5W2LVHZYDAHXNLQX32XX7IQ, seg_pos={0 0}, upload=476378]: Done uploading piece. err: <nil>
Dec 28 06:42:39.978193200 (543h13m39.849276245s) [noise=true , address=77.90.31.15:4700 , piece_id=ETRSZPCANUZ4RQKBF24SDMQGROXLA5AMUZ67ADFAWMZR4J2RNNCQ, seg_pos={0 0}, upload=476378]: Done uploading piece. err: <nil>
Dec 28 06:42:40.101092712 (543h13m39.972175757s) [noise=true , address=77.90.31.14:4661 , piece_id=ZDUE5DASOPITW7Z47QGXZEMX3WGB3KV2UAAJ7RGGTUXSJ27BR3BQ, seg_pos={0 0}, upload=476378]: Done uploading piece. err: <nil>
Dec 28 06:42:40.101142406 (543h13m39.972225441s) [seg_pos={0 0}, upload=476378]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:42:40.101374228 (543h13m39.972457273s) [noise=true , address=145.79.227.35:4482 , piece_id=FGLVIP7XPNOLQWS7BNG6YU2IL25QNDNMMYPHZEW3I32IEHKVAA4A, seg_pos={0 0}, upload=476378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Dec 28 06:42:40.101382233 (543h13m39.972465268s) [noise=true , address=145.79.227.35:4482 , piece_id=FGLVIP7XPNOLQWS7BNG6YU2IL25QNDNMMYPHZEW3I32IEHKVAA4A, seg_pos={0 0}, upload=476378]: Long tail cancellation due to optimal threshold
Dec 28 06:42:40.101450262 (543h13m39.972533297s) [noise=true , address=45.91.116.11:4118 , piece_id=UJH4KP52P4BTYVTOOAZ4P6MYZKLUNASNJAOIP7C3NQIZONAVXHCA, seg_pos={0 0}, upload=476378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:60986->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:60986->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60986->45.91.116.11:4118: use of closed network connection
Dec 28 06:42:40.101461384 (543h13m39.972544429s) [noise=true , address=45.91.116.11:4118 , piece_id=UJH4KP52P4BTYVTOOAZ4P6MYZKLUNASNJAOIP7C3NQIZONAVXHCA, seg_pos={0 0}, upload=476378]: Long tail cancellation due to optimal threshold
Dec 28 06:42:40.101489577 (543h13m39.972572612s) [noise=true , address=45.91.116.7:4011 , piece_id=QE35VFMSAK56VJFON5AYNICZIPJISO5FK5AAVABJVRX2HP2FQMWA, seg_pos={0 0}, upload=476378]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: write tcp 188.166.225.170:37250->45.91.116.7:4011: use of closed network connection; write tcp 188.166.225.170:37250->45.91.116.7:4011: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37250->45.91.116.7:4011: use of closed network connection
Dec 28 06:42:40.101528140 (543h13m39.972611185s) [noise=true , address=45.91.116.7:4011 , piece_id=QE35VFMSAK56VJFON5AYNICZIPJISO5FK5AAVABJVRX2HP2FQMWA, seg_pos={0 0}, upload=476378]: Long tail cancellation due to optimal threshold
Dec 28 06:42:40.101551175 (543h13m39.97263422s) [seg_pos={0 0}, upload=476378]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:42:40.101641327 (543h13m39.972724372s) [upload=476378]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:42:40.101647608 (543h13m39.972730653s) [upload=476378]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:42:40.390787721 (543h13m40.261870766s) [upload=476378]: Done uploading segments...
Dec 28 06:42:40.390811256 (543h13m40.261894301s) [upload=476378]: Upload finished. err: <nil>
Duration: 30.003684809s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
14 out of 60 is failed (threshold=6) [XX..X...X..XX.X...XX.....X................X.X.....X.X.......] duration:
4.175414906s trace:
https://jaeger.dev.storj.io/trace/6af3578081c62393
FAIL (2025-12-28 06:42:57 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.175435845s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [....X...X..X.XX.XXX..X..X.........X.XX..X....X.XX.........X.] duration:
5.883544469s trace:
https://jaeger.dev.storj.io/trace/2f6d89e322a3dde
FAIL (2025-12-28 06:42:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: CASE2LAI3QCHNEC4RYLV2N2AWZCNQZ5ISPNIIAL5XM6PKG7T4FRA) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: GIZYBAEUC5GQPDHMEINVZQH3US3PZGCWJ44JBDZM55LT65Y2E7TA) context deadline exceeded
======LOGS======
Dec 28 06:41:29.367275375 (543h12m29.23835842s) [upload=476361]: Starting upload
Dec 28 06:41:29.367347593 (543h12m29.238430628s) [upload=476361]: Done starting upload
Dec 28 06:41:29.367413538 (543h12m29.238496583s) [upload=476361]: Uploading segments...
Dec 28 06:41:29.367608739 (543h12m29.238691784s) [upload=476361]: Got next segment. Inline: false
Dec 28 06:41:29.367664426 (543h12m29.238747461s) [upload=476361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:41:29.367673242 (543h12m29.238756287s) [upload=476361]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:41:29.612140489 (543h12m29.483223555s) [seg_pos={0 0}, upload=476361]: Begin upload segment...
Dec 28 06:41:29.612199041 (543h12m29.483282086s) [seg_pos={0 0}, upload=476361]: Done begin upload segment.
Dec 28 06:41:29.612253485 (543h12m29.48333653s) [noise=true , address=145.79.227.31:4797 , piece_id=VQSSUQBH5E4ER5TFDOLOK4E6APFIWEW4FOWEZXWJI4GB4ZXPXRTQ, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.612285927 (543h12m29.483368962s) [noise=true , address=145.79.227.42:4123 , piece_id=PRU4PW2XDIDZKKMKWQZ53ACEUU4N6GKAPW6MUUCDIJBM5SGGSCJA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.612745432 (543h12m29.483828467s) [noise=true , address=77.90.31.16:4736 , piece_id=6BWRVNWHA6BXPJU5OLURHOR5LGCCL2UYHF7S5HHUEMLHFIXGALOQ, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.613036275 (543h12m29.48411931s) [noise=true , address=77.90.31.11:4553 , piece_id=EV4OG7Z2JUQAEGBI4E6VTNY6TL2B7K5Z53BRLXA5GDGJ535CXWHA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.613556115 (543h12m29.48463916s) [noise=true , address=145.79.227.15:4873 , piece_id=2HY2CWYVH3WLAQABM3CPPIUBWXNNSTDP3ABA37SBBAIWMTU7C3OA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.614044425 (543h12m29.48512748s) [noise=true , address=145.79.227.19:4079 , piece_id=VKFINQPWGQLG4QMPZWB5RUSORAXHMLLE77AF2EW3T35S5V7F67FA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.614626724 (543h12m29.485709779s) [noise=true , address=77.90.31.12:4589 , piece_id=NFXEVF4HNCTPGTDLYJCWYSRFPFX2JCSGMAFPMEWFPTDCMW27QOIA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.615130212 (543h12m29.486213258s) [noise=true , address=77.90.31.15:4697 , piece_id=CASE2LAI3QCHNEC4RYLV2N2AWZCNQZ5ISPNIIAL5XM6PKG7T4FRA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.616767600 (543h12m29.487850645s) [noise=true , address=145.79.227.24:4583 , piece_id=SPRO53EA2LUGBCRQAGH432OD7UPAE3G45AFJFVTXEK3Q7LA6MQPQ, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.618468569 (543h12m29.489551614s) [noise=true , address=45.91.116.12:4226 , piece_id=GIZYBAEUC5GQPDHMEINVZQH3US3PZGCWJ44JBDZM55LT65Y2E7TA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.620034211 (543h12m29.491117256s) [noise=true , address=45.91.116.10:4084 , piece_id=4KOROEPK54PJUZ7GSBEAOZODZYCR7G7IMRMO6T5AZGVRS4ZZZHDQ, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.621313817 (543h12m29.492396853s) [noise=true , address=145.79.227.2:4115 , piece_id=2STBIZRCMOHHOQ7NMZH4FW4OQHYVSK4KUMYZ6G74SPNRGWWH6BDA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.622777004 (543h12m29.493860049s) [noise=true , address=145.79.227.46:4602 , piece_id=XK5EZ5QA3SG7NTYL6KTITAS2Y34F2VUE222MHEX7MZGW57LCBOXA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.623607275 (543h12m29.49469032s) [noise=true , address=145.79.227.23:4547 , piece_id=KTJPR3WHS4XTP2A33LLPAR37BMJN7CWRWAQ73U2GDHMOKW5IOVFA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.625274139 (543h12m29.496357184s) [noise=true , address=145.79.227.47:4267 , piece_id=HL4Y3PE4P7MLFCFW476RFI3PAI23USHMKXFIY7EW2L2SYQ44E6DQ, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.625890552 (543h12m29.496973597s) [noise=true , address=145.79.227.45:4231 , piece_id=K7RK7PIWX65HRNWYX7GCQSF5TTEHRI53DXABBQW5UFH6EJDHANJA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:31.332822605 (543h12m31.20390565s) [upload=476361]: Next returned nil segment
Dec 28 06:41:31.332842553 (543h12m31.203925598s) [upload=476361]: Waiting for error group managing segments...
Dec 28 06:41:31.617149863 (543h12m31.488232908s) [noise=true , address=145.79.227.24:4583 , piece_id=SPRO53EA2LUGBCRQAGH432OD7UPAE3G45AFJFVTXEK3Q7LA6MQPQ, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.624304015 (543h12m31.49538705s) [noise=true , address=145.79.227.31:4797 , piece_id=VQSSUQBH5E4ER5TFDOLOK4E6APFIWEW4FOWEZXWJI4GB4ZXPXRTQ, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.631739534 (543h12m31.502822569s) [noise=true , address=145.79.227.45:4231 , piece_id=K7RK7PIWX65HRNWYX7GCQSF5TTEHRI53DXABBQW5UFH6EJDHANJA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.643464881 (543h12m31.514547916s) [noise=true , address=145.79.227.15:4873 , piece_id=2HY2CWYVH3WLAQABM3CPPIUBWXNNSTDP3ABA37SBBAIWMTU7C3OA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.644434617 (543h12m31.515517662s) [noise=true , address=145.79.227.19:4079 , piece_id=VKFINQPWGQLG4QMPZWB5RUSORAXHMLLE77AF2EW3T35S5V7F67FA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.647940052 (543h12m31.519023087s) [noise=true , address=145.79.227.47:4267 , piece_id=HL4Y3PE4P7MLFCFW476RFI3PAI23USHMKXFIY7EW2L2SYQ44E6DQ, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.652209422 (543h12m31.523292457s) [noise=true , address=145.79.227.23:4547 , piece_id=KTJPR3WHS4XTP2A33LLPAR37BMJN7CWRWAQ73U2GDHMOKW5IOVFA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.657955653 (543h12m31.529038698s) [noise=true , address=145.79.227.42:4123 , piece_id=PRU4PW2XDIDZKKMKWQZ53ACEUU4N6GKAPW6MUUCDIJBM5SGGSCJA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.660986093 (543h12m31.532069128s) [noise=true , address=145.79.227.2:4115 , piece_id=2STBIZRCMOHHOQ7NMZH4FW4OQHYVSK4KUMYZ6G74SPNRGWWH6BDA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.677360673 (543h12m31.548443718s) [noise=true , address=145.79.227.46:4602 , piece_id=XK5EZ5QA3SG7NTYL6KTITAS2Y34F2VUE222MHEX7MZGW57LCBOXA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.843236710 (543h12m31.714319745s) [noise=true , address=77.90.31.15:4697 , piece_id=CASE2LAI3QCHNEC4RYLV2N2AWZCNQZ5ISPNIIAL5XM6PKG7T4FRA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.866931599 (543h12m31.738014634s) [noise=true , address=45.91.116.12:4226 , piece_id=GIZYBAEUC5GQPDHMEINVZQH3US3PZGCWJ44JBDZM55LT65Y2E7TA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.886786246 (543h12m31.757869281s) [noise=true , address=77.90.31.12:4589 , piece_id=NFXEVF4HNCTPGTDLYJCWYSRFPFX2JCSGMAFPMEWFPTDCMW27QOIA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.886807867 (543h12m31.757890902s) [seg_pos={0 0}, upload=476361]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:41:31.887142585 (543h12m31.75822563s) [noise=true , address=77.90.31.16:4736 , piece_id=6BWRVNWHA6BXPJU5OLURHOR5LGCCL2UYHF7S5HHUEMLHFIXGALOQ, seg_pos={0 0}, upload=476361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 28 06:41:31.887537366 (543h12m31.758620411s) [noise=true , address=77.90.31.16:4736 , piece_id=6BWRVNWHA6BXPJU5OLURHOR5LGCCL2UYHF7S5HHUEMLHFIXGALOQ, seg_pos={0 0}, upload=476361]: Long tail cancellation due to optimal threshold
Dec 28 06:41:31.887598282 (543h12m31.758681327s) [noise=true , address=77.90.31.11:4553 , piece_id=EV4OG7Z2JUQAEGBI4E6VTNY6TL2B7K5Z53BRLXA5GDGJ535CXWHA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:38732->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:38732->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38732->77.90.31.11:4553: use of closed network connection
Dec 28 06:41:31.887606558 (543h12m31.758689593s) [noise=true , address=77.90.31.11:4553 , piece_id=EV4OG7Z2JUQAEGBI4E6VTNY6TL2B7K5Z53BRLXA5GDGJ535CXWHA, seg_pos={0 0}, upload=476361]: Long tail cancellation due to optimal threshold
Dec 28 06:41:31.887618350 (543h12m31.758701385s) [noise=true , address=45.91.116.10:4084 , piece_id=4KOROEPK54PJUZ7GSBEAOZODZYCR7G7IMRMO6T5AZGVRS4ZZZHDQ, seg_pos={0 0}, upload=476361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 28 06:41:31.887623430 (543h12m31.758706476s) [noise=true , address=45.91.116.10:4084 , piece_id=4KOROEPK54PJUZ7GSBEAOZODZYCR7G7IMRMO6T5AZGVRS4ZZZHDQ, seg_pos={0 0}, upload=476361]: Long tail cancellation due to optimal threshold
Dec 28 06:41:31.887641705 (543h12m31.75872474s) [seg_pos={0 0}, upload=476361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:41:31.887713471 (543h12m31.758796506s) [upload=476361]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:41:31.887718581 (543h12m31.758801616s) [upload=476361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:41:32.171820239 (543h12m32.042903304s) [upload=476361]: Done uploading segments...
Dec 28 06:41:32.171848763 (543h12m32.042931798s) [upload=476361]: Upload finished. err: <nil>
Duration: 5.883636933s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
14 out of 60 is failed (threshold=6) [.XX..X...X..XX.X...XX.....X................X.X.....X.X......] duration:
5.000660354s trace:
https://jaeger.dev.storj.io/trace/6af3578081c62393
FAIL (2025-12-28 06:41:59 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000716841s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X....X...X..X.XX.XXX..X..X.........X.XX..X....X.XX.........X] duration:
30.002471049s trace:
https://jaeger.dev.storj.io/trace/2f6d89e322a3dde
FAIL (2025-12-28 06:41:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: CASE2LAI3QCHNEC4RYLV2N2AWZCNQZ5ISPNIIAL5XM6PKG7T4FRA) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: GIZYBAEUC5GQPDHMEINVZQH3US3PZGCWJ44JBDZM55LT65Y2E7TA) context deadline exceeded
======LOGS======
Dec 28 06:41:29.367275375 (543h12m29.23835842s) [upload=476361]: Starting upload
Dec 28 06:41:29.367347593 (543h12m29.238430628s) [upload=476361]: Done starting upload
Dec 28 06:41:29.367413538 (543h12m29.238496583s) [upload=476361]: Uploading segments...
Dec 28 06:41:29.367608739 (543h12m29.238691784s) [upload=476361]: Got next segment. Inline: false
Dec 28 06:41:29.367664426 (543h12m29.238747461s) [upload=476361]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:41:29.367673242 (543h12m29.238756287s) [upload=476361]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:41:29.612140489 (543h12m29.483223555s) [seg_pos={0 0}, upload=476361]: Begin upload segment...
Dec 28 06:41:29.612199041 (543h12m29.483282086s) [seg_pos={0 0}, upload=476361]: Done begin upload segment.
Dec 28 06:41:29.612253485 (543h12m29.48333653s) [noise=true , address=145.79.227.31:4797 , piece_id=VQSSUQBH5E4ER5TFDOLOK4E6APFIWEW4FOWEZXWJI4GB4ZXPXRTQ, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.612285927 (543h12m29.483368962s) [noise=true , address=145.79.227.42:4123 , piece_id=PRU4PW2XDIDZKKMKWQZ53ACEUU4N6GKAPW6MUUCDIJBM5SGGSCJA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.612745432 (543h12m29.483828467s) [noise=true , address=77.90.31.16:4736 , piece_id=6BWRVNWHA6BXPJU5OLURHOR5LGCCL2UYHF7S5HHUEMLHFIXGALOQ, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.613036275 (543h12m29.48411931s) [noise=true , address=77.90.31.11:4553 , piece_id=EV4OG7Z2JUQAEGBI4E6VTNY6TL2B7K5Z53BRLXA5GDGJ535CXWHA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.613556115 (543h12m29.48463916s) [noise=true , address=145.79.227.15:4873 , piece_id=2HY2CWYVH3WLAQABM3CPPIUBWXNNSTDP3ABA37SBBAIWMTU7C3OA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.614044425 (543h12m29.48512748s) [noise=true , address=145.79.227.19:4079 , piece_id=VKFINQPWGQLG4QMPZWB5RUSORAXHMLLE77AF2EW3T35S5V7F67FA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.614626724 (543h12m29.485709779s) [noise=true , address=77.90.31.12:4589 , piece_id=NFXEVF4HNCTPGTDLYJCWYSRFPFX2JCSGMAFPMEWFPTDCMW27QOIA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.615130212 (543h12m29.486213258s) [noise=true , address=77.90.31.15:4697 , piece_id=CASE2LAI3QCHNEC4RYLV2N2AWZCNQZ5ISPNIIAL5XM6PKG7T4FRA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.616767600 (543h12m29.487850645s) [noise=true , address=145.79.227.24:4583 , piece_id=SPRO53EA2LUGBCRQAGH432OD7UPAE3G45AFJFVTXEK3Q7LA6MQPQ, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.618468569 (543h12m29.489551614s) [noise=true , address=45.91.116.12:4226 , piece_id=GIZYBAEUC5GQPDHMEINVZQH3US3PZGCWJ44JBDZM55LT65Y2E7TA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.620034211 (543h12m29.491117256s) [noise=true , address=45.91.116.10:4084 , piece_id=4KOROEPK54PJUZ7GSBEAOZODZYCR7G7IMRMO6T5AZGVRS4ZZZHDQ, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.621313817 (543h12m29.492396853s) [noise=true , address=145.79.227.2:4115 , piece_id=2STBIZRCMOHHOQ7NMZH4FW4OQHYVSK4KUMYZ6G74SPNRGWWH6BDA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.622777004 (543h12m29.493860049s) [noise=true , address=145.79.227.46:4602 , piece_id=XK5EZ5QA3SG7NTYL6KTITAS2Y34F2VUE222MHEX7MZGW57LCBOXA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.623607275 (543h12m29.49469032s) [noise=true , address=145.79.227.23:4547 , piece_id=KTJPR3WHS4XTP2A33LLPAR37BMJN7CWRWAQ73U2GDHMOKW5IOVFA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.625274139 (543h12m29.496357184s) [noise=true , address=145.79.227.47:4267 , piece_id=HL4Y3PE4P7MLFCFW476RFI3PAI23USHMKXFIY7EW2L2SYQ44E6DQ, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:29.625890552 (543h12m29.496973597s) [noise=true , address=145.79.227.45:4231 , piece_id=K7RK7PIWX65HRNWYX7GCQSF5TTEHRI53DXABBQW5UFH6EJDHANJA, seg_pos={0 0}, upload=476361]: Uploading piece...
Dec 28 06:41:31.332822605 (543h12m31.20390565s) [upload=476361]: Next returned nil segment
Dec 28 06:41:31.332842553 (543h12m31.203925598s) [upload=476361]: Waiting for error group managing segments...
Dec 28 06:41:31.617149863 (543h12m31.488232908s) [noise=true , address=145.79.227.24:4583 , piece_id=SPRO53EA2LUGBCRQAGH432OD7UPAE3G45AFJFVTXEK3Q7LA6MQPQ, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.624304015 (543h12m31.49538705s) [noise=true , address=145.79.227.31:4797 , piece_id=VQSSUQBH5E4ER5TFDOLOK4E6APFIWEW4FOWEZXWJI4GB4ZXPXRTQ, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.631739534 (543h12m31.502822569s) [noise=true , address=145.79.227.45:4231 , piece_id=K7RK7PIWX65HRNWYX7GCQSF5TTEHRI53DXABBQW5UFH6EJDHANJA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.643464881 (543h12m31.514547916s) [noise=true , address=145.79.227.15:4873 , piece_id=2HY2CWYVH3WLAQABM3CPPIUBWXNNSTDP3ABA37SBBAIWMTU7C3OA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.644434617 (543h12m31.515517662s) [noise=true , address=145.79.227.19:4079 , piece_id=VKFINQPWGQLG4QMPZWB5RUSORAXHMLLE77AF2EW3T35S5V7F67FA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.647940052 (543h12m31.519023087s) [noise=true , address=145.79.227.47:4267 , piece_id=HL4Y3PE4P7MLFCFW476RFI3PAI23USHMKXFIY7EW2L2SYQ44E6DQ, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.652209422 (543h12m31.523292457s) [noise=true , address=145.79.227.23:4547 , piece_id=KTJPR3WHS4XTP2A33LLPAR37BMJN7CWRWAQ73U2GDHMOKW5IOVFA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.657955653 (543h12m31.529038698s) [noise=true , address=145.79.227.42:4123 , piece_id=PRU4PW2XDIDZKKMKWQZ53ACEUU4N6GKAPW6MUUCDIJBM5SGGSCJA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.660986093 (543h12m31.532069128s) [noise=true , address=145.79.227.2:4115 , piece_id=2STBIZRCMOHHOQ7NMZH4FW4OQHYVSK4KUMYZ6G74SPNRGWWH6BDA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.677360673 (543h12m31.548443718s) [noise=true , address=145.79.227.46:4602 , piece_id=XK5EZ5QA3SG7NTYL6KTITAS2Y34F2VUE222MHEX7MZGW57LCBOXA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.843236710 (543h12m31.714319745s) [noise=true , address=77.90.31.15:4697 , piece_id=CASE2LAI3QCHNEC4RYLV2N2AWZCNQZ5ISPNIIAL5XM6PKG7T4FRA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.866931599 (543h12m31.738014634s) [noise=true , address=45.91.116.12:4226 , piece_id=GIZYBAEUC5GQPDHMEINVZQH3US3PZGCWJ44JBDZM55LT65Y2E7TA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.886786246 (543h12m31.757869281s) [noise=true , address=77.90.31.12:4589 , piece_id=NFXEVF4HNCTPGTDLYJCWYSRFPFX2JCSGMAFPMEWFPTDCMW27QOIA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: <nil>
Dec 28 06:41:31.886807867 (543h12m31.757890902s) [seg_pos={0 0}, upload=476361]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:41:31.887142585 (543h12m31.75822563s) [noise=true , address=77.90.31.16:4736 , piece_id=6BWRVNWHA6BXPJU5OLURHOR5LGCCL2UYHF7S5HHUEMLHFIXGALOQ, seg_pos={0 0}, upload=476361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 28 06:41:31.887537366 (543h12m31.758620411s) [noise=true , address=77.90.31.16:4736 , piece_id=6BWRVNWHA6BXPJU5OLURHOR5LGCCL2UYHF7S5HHUEMLHFIXGALOQ, seg_pos={0 0}, upload=476361]: Long tail cancellation due to optimal threshold
Dec 28 06:41:31.887598282 (543h12m31.758681327s) [noise=true , address=77.90.31.11:4553 , piece_id=EV4OG7Z2JUQAEGBI4E6VTNY6TL2B7K5Z53BRLXA5GDGJ535CXWHA, seg_pos={0 0}, upload=476361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:38732->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:38732->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38732->77.90.31.11:4553: use of closed network connection
Dec 28 06:41:31.887606558 (543h12m31.758689593s) [noise=true , address=77.90.31.11:4553 , piece_id=EV4OG7Z2JUQAEGBI4E6VTNY6TL2B7K5Z53BRLXA5GDGJ535CXWHA, seg_pos={0 0}, upload=476361]: Long tail cancellation due to optimal threshold
Dec 28 06:41:31.887618350 (543h12m31.758701385s) [noise=true , address=45.91.116.10:4084 , piece_id=4KOROEPK54PJUZ7GSBEAOZODZYCR7G7IMRMO6T5AZGVRS4ZZZHDQ, seg_pos={0 0}, upload=476361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Dec 28 06:41:31.887623430 (543h12m31.758706476s) [noise=true , address=45.91.116.10:4084 , piece_id=4KOROEPK54PJUZ7GSBEAOZODZYCR7G7IMRMO6T5AZGVRS4ZZZHDQ, seg_pos={0 0}, upload=476361]: Long tail cancellation due to optimal threshold
Dec 28 06:41:31.887641705 (543h12m31.75872474s) [seg_pos={0 0}, upload=476361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:41:31.887713471 (543h12m31.758796506s) [upload=476361]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:41:31.887718581 (543h12m31.758801616s) [upload=476361]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:41:32.171820239 (543h12m32.042903304s) [upload=476361]: Done uploading segments...
Dec 28 06:41:32.171848763 (543h12m32.042931798s) [upload=476361]: Upload finished. err: <nil>
Duration: 30.002504431s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.X....X...X..X.XX.XXX..X..X.........X.XX..X....X.XX.........] duration:
5.733672945s trace:
https://jaeger.dev.storj.io/trace/704c6935d5cc4092
FAIL (2025-12-28 06:41:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
h5m9.486928352s) [upload=476257]: Got next segment. Inline: false
Dec 28 06:34:09.615898808 (543h5m9.486981843s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:34:09.615905691 (543h5m9.486988726s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:34:09.865608412 (543h5m9.736691457s) [seg_pos={0 0}, upload=476257]: Begin upload segment...
Dec 28 06:34:09.865736285 (543h5m9.73681933s) [noise=true , address=45.91.116.13:4153 , piece_id=BZMRSMICRJUIEDMJHWSI67BQPNPWZAUNBJ3CX5CDGL3FYRD6PMGA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.865749070 (543h5m9.736832125s) [seg_pos={0 0}, upload=476257]: Done begin upload segment.
Dec 28 06:34:09.865787202 (543h5m9.736870247s) [noise=true , address=145.79.227.31:4797 , piece_id=Y5G3XZWKUKJWO4VJ6VVK2KPJPL6DBUNBL2FKDH5KN4TKSYBD6JUA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.866818415 (543h5m9.73790145s) [noise=true , address=145.79.227.45:4229 , piece_id=YTSYDZTA5NPFG7RJD5QAUVBONIVZIYXDLVCSDGFJH2Z3RYBQKR6Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.867255428 (543h5m9.738338473s) [noise=true , address=45.91.116.14:4191 , piece_id=FSGCS4I2W6AXKBAAKIO3H3VVDJCMO2ZJC5I7UTPL3HPWEGFPCH4A, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.867793983 (543h5m9.738877028s) [noise=true , address=77.90.31.19:4842 , piece_id=EPUFZMC6JEEICGDHMTIJ33326OS3Q5LPK6BYCS2MCSLHH4RI2JQQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868206017 (543h5m9.739289052s) [noise=true , address=77.90.31.11:4553 , piece_id=36TL2YGUG55F3AWIBZOLH4NWEOFQAMAFIYI5DYHRZOA7C67AU2BA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868217238 (543h5m9.739300284s) [noise=true , address=145.79.227.11:4729 , piece_id=NOQ6KFZAFIPCILD6R4IIZPADYTODHLMKUCOZOYF6AHFWB3B5CRTA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868668868 (543h5m9.739751904s) [noise=true , address=77.90.31.14:4663 , piece_id=JWAWZOGO35BGX4UXNU7Q726YXBEANHX5T4PITTNXLKSS5FYDXHIA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868732500 (543h5m9.739815535s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869147069 (543h5m9.740230114s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869179722 (543h5m9.740262767s) [noise=true , address=145.79.227.25:4619 , piece_id=QE4ISVK6E4HAJ66OODOSO6BAQBP7BETNWQRU6YYU4J7WW5MK2WGA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869611795 (543h5m9.74069484s) [noise=true , address=145.79.227.16:4911 , piece_id=2YZ4IVX7IEVT2NMS5DPF7VAOYRJFZX2BW2NQ4KCTJG4DS4KS462Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.870118559 (543h5m9.741201604s) [noise=true , address=45.91.116.9:4263 , piece_id=O7RGLAZBP7JGULRDJLYI3F65OMR5BQJDHJBF5YJJY6EVQHDCQH7Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.870684617 (543h5m9.741767672s) [noise=true , address=145.79.227.50:4375 , piece_id=K5GHFVXIHUTT3IQLDKFFFYKVZ24MFJR2RMQVNKKI4BIQW4ZLHEMQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.871095760 (543h5m9.742178805s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.872053514 (543h5m9.743136549s) [noise=true , address=145.79.227.17:4972 , piece_id=WLEHEL6HUTMH4DXZJC3IDYNS3WMVEMILDDSAPL2HQO6SQRJCILFQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:11.740792204 (543h5m11.611875239s) [upload=476257]: Next returned nil segment
Dec 28 06:34:11.740807092 (543h5m11.611890127s) [upload=476257]: Waiting for error group managing segments...
Dec 28 06:34:12.077641418 (543h5m11.948724453s) [noise=true , address=77.90.31.19:4842 , piece_id=EPUFZMC6JEEICGDHMTIJ33326OS3Q5LPK6BYCS2MCSLHH4RI2JQQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.140083399 (543h5m12.011166434s) [noise=true , address=45.91.116.13:4153 , piece_id=BZMRSMICRJUIEDMJHWSI67BQPNPWZAUNBJ3CX5CDGL3FYRD6PMGA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.247241838 (543h5m12.118324883s) [noise=true , address=45.91.116.14:4191 , piece_id=FSGCS4I2W6AXKBAAKIO3H3VVDJCMO2ZJC5I7UTPL3HPWEGFPCH4A, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.281101414 (543h5m12.152184459s) [noise=true , address=77.90.31.11:4553 , piece_id=36TL2YGUG55F3AWIBZOLH4NWEOFQAMAFIYI5DYHRZOA7C67AU2BA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.572990687 (543h5m12.444073732s) [noise=true , address=45.91.116.9:4263 , piece_id=O7RGLAZBP7JGULRDJLYI3F65OMR5BQJDHJBF5YJJY6EVQHDCQH7Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.700970623 (543h5m12.572053658s) [noise=true , address=145.79.227.11:4729 , piece_id=NOQ6KFZAFIPCILD6R4IIZPADYTODHLMKUCOZOYF6AHFWB3B5CRTA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.722390811 (543h5m12.593473846s) [noise=true , address=145.79.227.31:4797 , piece_id=Y5G3XZWKUKJWO4VJ6VVK2KPJPL6DBUNBL2FKDH5KN4TKSYBD6JUA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.751548203 (543h5m12.622631248s) [noise=true , address=145.79.227.45:4229 , piece_id=YTSYDZTA5NPFG7RJD5QAUVBONIVZIYXDLVCSDGFJH2Z3RYBQKR6Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.086115831 (543h5m12.957198866s) [noise=true , address=77.90.31.14:4663 , piece_id=JWAWZOGO35BGX4UXNU7Q726YXBEANHX5T4PITTNXLKSS5FYDXHIA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.088302755 (543h5m12.959385801s) [noise=true , address=145.79.227.50:4375 , piece_id=K5GHFVXIHUTT3IQLDKFFFYKVZ24MFJR2RMQVNKKI4BIQW4ZLHEMQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.163593835 (543h5m13.03467687s) [noise=true , address=145.79.227.25:4619 , piece_id=QE4ISVK6E4HAJ66OODOSO6BAQBP7BETNWQRU6YYU4J7WW5MK2WGA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.180506259 (543h5m13.051589304s) [noise=true , address=145.79.227.17:4972 , piece_id=WLEHEL6HUTMH4DXZJC3IDYNS3WMVEMILDDSAPL2HQO6SQRJCILFQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.280406527 (543h5m13.151489562s) [noise=true , address=145.79.227.16:4911 , piece_id=2YZ4IVX7IEVT2NMS5DPF7VAOYRJFZX2BW2NQ4KCTJG4DS4KS462Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.280435773 (543h5m13.151518818s) [seg_pos={0 0}, upload=476257]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:34:13.280623320 (543h5m13.151706355s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection
Dec 28 06:34:13.280628881 (543h5m13.151711916s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.280789476 (543h5m13.151872511s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection
Dec 28 06:34:13.280804916 (543h5m13.151887961s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.281055182 (543h5m13.152138227s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7): protocol: read tcp 188.166.225.170:55658->145.79.227.52:4446: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 06:34:13.281065091 (543h5m13.152148126s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.281088385 (543h5m13.152171431s) [seg_pos={0 0}, upload=476257]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:34:13.281175762 (543h5m13.152258807s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:34:13.281182455 (543h5m13.15226549s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:34:13.580509800 (543h5m13.451592845s) [upload=476257]: Done uploading segments...
Dec 28 06:34:13.580540228 (543h5m13.451623293s) [upload=476257]: Upload finished. err: <nil>
Duration: 5.733701759s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [X.XX..X...X..XX.X...XX.....X................X.X.....X.X.....] duration:
4.933080932s trace:
https://jaeger.dev.storj.io/trace/6af3578081c62393
FAIL (2025-12-28 06:41:12 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.933108555s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [..X....X...X..X.XX.XXX..X..X.........X.XX..X....X.XX........] duration:
28.434895638s trace:
https://jaeger.dev.storj.io/trace/704c6935d5cc4092
FAIL (2025-12-28 06:41:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
h5m9.486928352s) [upload=476257]: Got next segment. Inline: false
Dec 28 06:34:09.615898808 (543h5m9.486981843s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:34:09.615905691 (543h5m9.486988726s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:34:09.865608412 (543h5m9.736691457s) [seg_pos={0 0}, upload=476257]: Begin upload segment...
Dec 28 06:34:09.865736285 (543h5m9.73681933s) [noise=true , address=45.91.116.13:4153 , piece_id=BZMRSMICRJUIEDMJHWSI67BQPNPWZAUNBJ3CX5CDGL3FYRD6PMGA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.865749070 (543h5m9.736832125s) [seg_pos={0 0}, upload=476257]: Done begin upload segment.
Dec 28 06:34:09.865787202 (543h5m9.736870247s) [noise=true , address=145.79.227.31:4797 , piece_id=Y5G3XZWKUKJWO4VJ6VVK2KPJPL6DBUNBL2FKDH5KN4TKSYBD6JUA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.866818415 (543h5m9.73790145s) [noise=true , address=145.79.227.45:4229 , piece_id=YTSYDZTA5NPFG7RJD5QAUVBONIVZIYXDLVCSDGFJH2Z3RYBQKR6Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.867255428 (543h5m9.738338473s) [noise=true , address=45.91.116.14:4191 , piece_id=FSGCS4I2W6AXKBAAKIO3H3VVDJCMO2ZJC5I7UTPL3HPWEGFPCH4A, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.867793983 (543h5m9.738877028s) [noise=true , address=77.90.31.19:4842 , piece_id=EPUFZMC6JEEICGDHMTIJ33326OS3Q5LPK6BYCS2MCSLHH4RI2JQQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868206017 (543h5m9.739289052s) [noise=true , address=77.90.31.11:4553 , piece_id=36TL2YGUG55F3AWIBZOLH4NWEOFQAMAFIYI5DYHRZOA7C67AU2BA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868217238 (543h5m9.739300284s) [noise=true , address=145.79.227.11:4729 , piece_id=NOQ6KFZAFIPCILD6R4IIZPADYTODHLMKUCOZOYF6AHFWB3B5CRTA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868668868 (543h5m9.739751904s) [noise=true , address=77.90.31.14:4663 , piece_id=JWAWZOGO35BGX4UXNU7Q726YXBEANHX5T4PITTNXLKSS5FYDXHIA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868732500 (543h5m9.739815535s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869147069 (543h5m9.740230114s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869179722 (543h5m9.740262767s) [noise=true , address=145.79.227.25:4619 , piece_id=QE4ISVK6E4HAJ66OODOSO6BAQBP7BETNWQRU6YYU4J7WW5MK2WGA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869611795 (543h5m9.74069484s) [noise=true , address=145.79.227.16:4911 , piece_id=2YZ4IVX7IEVT2NMS5DPF7VAOYRJFZX2BW2NQ4KCTJG4DS4KS462Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.870118559 (543h5m9.741201604s) [noise=true , address=45.91.116.9:4263 , piece_id=O7RGLAZBP7JGULRDJLYI3F65OMR5BQJDHJBF5YJJY6EVQHDCQH7Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.870684617 (543h5m9.741767672s) [noise=true , address=145.79.227.50:4375 , piece_id=K5GHFVXIHUTT3IQLDKFFFYKVZ24MFJR2RMQVNKKI4BIQW4ZLHEMQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.871095760 (543h5m9.742178805s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.872053514 (543h5m9.743136549s) [noise=true , address=145.79.227.17:4972 , piece_id=WLEHEL6HUTMH4DXZJC3IDYNS3WMVEMILDDSAPL2HQO6SQRJCILFQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:11.740792204 (543h5m11.611875239s) [upload=476257]: Next returned nil segment
Dec 28 06:34:11.740807092 (543h5m11.611890127s) [upload=476257]: Waiting for error group managing segments...
Dec 28 06:34:12.077641418 (543h5m11.948724453s) [noise=true , address=77.90.31.19:4842 , piece_id=EPUFZMC6JEEICGDHMTIJ33326OS3Q5LPK6BYCS2MCSLHH4RI2JQQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.140083399 (543h5m12.011166434s) [noise=true , address=45.91.116.13:4153 , piece_id=BZMRSMICRJUIEDMJHWSI67BQPNPWZAUNBJ3CX5CDGL3FYRD6PMGA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.247241838 (543h5m12.118324883s) [noise=true , address=45.91.116.14:4191 , piece_id=FSGCS4I2W6AXKBAAKIO3H3VVDJCMO2ZJC5I7UTPL3HPWEGFPCH4A, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.281101414 (543h5m12.152184459s) [noise=true , address=77.90.31.11:4553 , piece_id=36TL2YGUG55F3AWIBZOLH4NWEOFQAMAFIYI5DYHRZOA7C67AU2BA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.572990687 (543h5m12.444073732s) [noise=true , address=45.91.116.9:4263 , piece_id=O7RGLAZBP7JGULRDJLYI3F65OMR5BQJDHJBF5YJJY6EVQHDCQH7Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.700970623 (543h5m12.572053658s) [noise=true , address=145.79.227.11:4729 , piece_id=NOQ6KFZAFIPCILD6R4IIZPADYTODHLMKUCOZOYF6AHFWB3B5CRTA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.722390811 (543h5m12.593473846s) [noise=true , address=145.79.227.31:4797 , piece_id=Y5G3XZWKUKJWO4VJ6VVK2KPJPL6DBUNBL2FKDH5KN4TKSYBD6JUA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.751548203 (543h5m12.622631248s) [noise=true , address=145.79.227.45:4229 , piece_id=YTSYDZTA5NPFG7RJD5QAUVBONIVZIYXDLVCSDGFJH2Z3RYBQKR6Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.086115831 (543h5m12.957198866s) [noise=true , address=77.90.31.14:4663 , piece_id=JWAWZOGO35BGX4UXNU7Q726YXBEANHX5T4PITTNXLKSS5FYDXHIA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.088302755 (543h5m12.959385801s) [noise=true , address=145.79.227.50:4375 , piece_id=K5GHFVXIHUTT3IQLDKFFFYKVZ24MFJR2RMQVNKKI4BIQW4ZLHEMQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.163593835 (543h5m13.03467687s) [noise=true , address=145.79.227.25:4619 , piece_id=QE4ISVK6E4HAJ66OODOSO6BAQBP7BETNWQRU6YYU4J7WW5MK2WGA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.180506259 (543h5m13.051589304s) [noise=true , address=145.79.227.17:4972 , piece_id=WLEHEL6HUTMH4DXZJC3IDYNS3WMVEMILDDSAPL2HQO6SQRJCILFQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.280406527 (543h5m13.151489562s) [noise=true , address=145.79.227.16:4911 , piece_id=2YZ4IVX7IEVT2NMS5DPF7VAOYRJFZX2BW2NQ4KCTJG4DS4KS462Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.280435773 (543h5m13.151518818s) [seg_pos={0 0}, upload=476257]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:34:13.280623320 (543h5m13.151706355s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection
Dec 28 06:34:13.280628881 (543h5m13.151711916s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.280789476 (543h5m13.151872511s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection
Dec 28 06:34:13.280804916 (543h5m13.151887961s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.281055182 (543h5m13.152138227s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7): protocol: read tcp 188.166.225.170:55658->145.79.227.52:4446: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 06:34:13.281065091 (543h5m13.152148126s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.281088385 (543h5m13.152171431s) [seg_pos={0 0}, upload=476257]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:34:13.281175762 (543h5m13.152258807s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:34:13.281182455 (543h5m13.15226549s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:34:13.580509800 (543h5m13.451592845s) [upload=476257]: Done uploading segments...
Dec 28 06:34:13.580540228 (543h5m13.451623293s) [upload=476257]: Upload finished. err: <nil>
Duration: 28.434916098s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
15 out of 60 is failed (threshold=6) [.X.XX..X...X..XX.X...XX.....X................X.X.....X.X....] duration:
5.001023294s trace:
https://jaeger.dev.storj.io/trace/6af3578081c62393
FAIL (2025-12-28 06:40:44 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001048824s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [X.X.XX..X...X..XX.X...XX.....X................X.X.....X.X...] duration:
5.00162467s trace:
https://jaeger.dev.storj.io/trace/6af3578081c62393
FAIL (2025-12-28 06:39:40 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001649398s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X..X....X...X..X.XX.XXX..X..X.........X.XX..X....X.XX.......] duration:
5.928349635s trace:
https://jaeger.dev.storj.io/trace/704c6935d5cc4092
FAIL (2025-12-28 06:39:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
h5m9.486928352s) [upload=476257]: Got next segment. Inline: false
Dec 28 06:34:09.615898808 (543h5m9.486981843s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:34:09.615905691 (543h5m9.486988726s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:34:09.865608412 (543h5m9.736691457s) [seg_pos={0 0}, upload=476257]: Begin upload segment...
Dec 28 06:34:09.865736285 (543h5m9.73681933s) [noise=true , address=45.91.116.13:4153 , piece_id=BZMRSMICRJUIEDMJHWSI67BQPNPWZAUNBJ3CX5CDGL3FYRD6PMGA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.865749070 (543h5m9.736832125s) [seg_pos={0 0}, upload=476257]: Done begin upload segment.
Dec 28 06:34:09.865787202 (543h5m9.736870247s) [noise=true , address=145.79.227.31:4797 , piece_id=Y5G3XZWKUKJWO4VJ6VVK2KPJPL6DBUNBL2FKDH5KN4TKSYBD6JUA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.866818415 (543h5m9.73790145s) [noise=true , address=145.79.227.45:4229 , piece_id=YTSYDZTA5NPFG7RJD5QAUVBONIVZIYXDLVCSDGFJH2Z3RYBQKR6Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.867255428 (543h5m9.738338473s) [noise=true , address=45.91.116.14:4191 , piece_id=FSGCS4I2W6AXKBAAKIO3H3VVDJCMO2ZJC5I7UTPL3HPWEGFPCH4A, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.867793983 (543h5m9.738877028s) [noise=true , address=77.90.31.19:4842 , piece_id=EPUFZMC6JEEICGDHMTIJ33326OS3Q5LPK6BYCS2MCSLHH4RI2JQQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868206017 (543h5m9.739289052s) [noise=true , address=77.90.31.11:4553 , piece_id=36TL2YGUG55F3AWIBZOLH4NWEOFQAMAFIYI5DYHRZOA7C67AU2BA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868217238 (543h5m9.739300284s) [noise=true , address=145.79.227.11:4729 , piece_id=NOQ6KFZAFIPCILD6R4IIZPADYTODHLMKUCOZOYF6AHFWB3B5CRTA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868668868 (543h5m9.739751904s) [noise=true , address=77.90.31.14:4663 , piece_id=JWAWZOGO35BGX4UXNU7Q726YXBEANHX5T4PITTNXLKSS5FYDXHIA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868732500 (543h5m9.739815535s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869147069 (543h5m9.740230114s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869179722 (543h5m9.740262767s) [noise=true , address=145.79.227.25:4619 , piece_id=QE4ISVK6E4HAJ66OODOSO6BAQBP7BETNWQRU6YYU4J7WW5MK2WGA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869611795 (543h5m9.74069484s) [noise=true , address=145.79.227.16:4911 , piece_id=2YZ4IVX7IEVT2NMS5DPF7VAOYRJFZX2BW2NQ4KCTJG4DS4KS462Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.870118559 (543h5m9.741201604s) [noise=true , address=45.91.116.9:4263 , piece_id=O7RGLAZBP7JGULRDJLYI3F65OMR5BQJDHJBF5YJJY6EVQHDCQH7Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.870684617 (543h5m9.741767672s) [noise=true , address=145.79.227.50:4375 , piece_id=K5GHFVXIHUTT3IQLDKFFFYKVZ24MFJR2RMQVNKKI4BIQW4ZLHEMQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.871095760 (543h5m9.742178805s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.872053514 (543h5m9.743136549s) [noise=true , address=145.79.227.17:4972 , piece_id=WLEHEL6HUTMH4DXZJC3IDYNS3WMVEMILDDSAPL2HQO6SQRJCILFQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:11.740792204 (543h5m11.611875239s) [upload=476257]: Next returned nil segment
Dec 28 06:34:11.740807092 (543h5m11.611890127s) [upload=476257]: Waiting for error group managing segments...
Dec 28 06:34:12.077641418 (543h5m11.948724453s) [noise=true , address=77.90.31.19:4842 , piece_id=EPUFZMC6JEEICGDHMTIJ33326OS3Q5LPK6BYCS2MCSLHH4RI2JQQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.140083399 (543h5m12.011166434s) [noise=true , address=45.91.116.13:4153 , piece_id=BZMRSMICRJUIEDMJHWSI67BQPNPWZAUNBJ3CX5CDGL3FYRD6PMGA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.247241838 (543h5m12.118324883s) [noise=true , address=45.91.116.14:4191 , piece_id=FSGCS4I2W6AXKBAAKIO3H3VVDJCMO2ZJC5I7UTPL3HPWEGFPCH4A, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.281101414 (543h5m12.152184459s) [noise=true , address=77.90.31.11:4553 , piece_id=36TL2YGUG55F3AWIBZOLH4NWEOFQAMAFIYI5DYHRZOA7C67AU2BA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.572990687 (543h5m12.444073732s) [noise=true , address=45.91.116.9:4263 , piece_id=O7RGLAZBP7JGULRDJLYI3F65OMR5BQJDHJBF5YJJY6EVQHDCQH7Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.700970623 (543h5m12.572053658s) [noise=true , address=145.79.227.11:4729 , piece_id=NOQ6KFZAFIPCILD6R4IIZPADYTODHLMKUCOZOYF6AHFWB3B5CRTA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.722390811 (543h5m12.593473846s) [noise=true , address=145.79.227.31:4797 , piece_id=Y5G3XZWKUKJWO4VJ6VVK2KPJPL6DBUNBL2FKDH5KN4TKSYBD6JUA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.751548203 (543h5m12.622631248s) [noise=true , address=145.79.227.45:4229 , piece_id=YTSYDZTA5NPFG7RJD5QAUVBONIVZIYXDLVCSDGFJH2Z3RYBQKR6Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.086115831 (543h5m12.957198866s) [noise=true , address=77.90.31.14:4663 , piece_id=JWAWZOGO35BGX4UXNU7Q726YXBEANHX5T4PITTNXLKSS5FYDXHIA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.088302755 (543h5m12.959385801s) [noise=true , address=145.79.227.50:4375 , piece_id=K5GHFVXIHUTT3IQLDKFFFYKVZ24MFJR2RMQVNKKI4BIQW4ZLHEMQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.163593835 (543h5m13.03467687s) [noise=true , address=145.79.227.25:4619 , piece_id=QE4ISVK6E4HAJ66OODOSO6BAQBP7BETNWQRU6YYU4J7WW5MK2WGA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.180506259 (543h5m13.051589304s) [noise=true , address=145.79.227.17:4972 , piece_id=WLEHEL6HUTMH4DXZJC3IDYNS3WMVEMILDDSAPL2HQO6SQRJCILFQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.280406527 (543h5m13.151489562s) [noise=true , address=145.79.227.16:4911 , piece_id=2YZ4IVX7IEVT2NMS5DPF7VAOYRJFZX2BW2NQ4KCTJG4DS4KS462Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.280435773 (543h5m13.151518818s) [seg_pos={0 0}, upload=476257]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:34:13.280623320 (543h5m13.151706355s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection
Dec 28 06:34:13.280628881 (543h5m13.151711916s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.280789476 (543h5m13.151872511s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection
Dec 28 06:34:13.280804916 (543h5m13.151887961s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.281055182 (543h5m13.152138227s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7): protocol: read tcp 188.166.225.170:55658->145.79.227.52:4446: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 06:34:13.281065091 (543h5m13.152148126s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.281088385 (543h5m13.152171431s) [seg_pos={0 0}, upload=476257]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:34:13.281175762 (543h5m13.152258807s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:34:13.281182455 (543h5m13.15226549s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:34:13.580509800 (543h5m13.451592845s) [upload=476257]: Done uploading segments...
Dec 28 06:34:13.580540228 (543h5m13.451623293s) [upload=476257]: Upload finished. err: <nil>
Duration: 5.928382768s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [.X.X.XX..X...X..XX.X...XX.....X................X.X.....X.X..] duration:
3.502367086s trace:
https://jaeger.dev.storj.io/trace/6af3578081c62393
FAIL (2025-12-28 06:39:10 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.50240085s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.X..X....X...X..X.XX.XXX..X..X.........X.XX..X....X.XX......] duration:
5.922311528s trace:
https://jaeger.dev.storj.io/trace/704c6935d5cc4092
FAIL (2025-12-28 06:38:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
h5m9.486928352s) [upload=476257]: Got next segment. Inline: false
Dec 28 06:34:09.615898808 (543h5m9.486981843s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:34:09.615905691 (543h5m9.486988726s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:34:09.865608412 (543h5m9.736691457s) [seg_pos={0 0}, upload=476257]: Begin upload segment...
Dec 28 06:34:09.865736285 (543h5m9.73681933s) [noise=true , address=45.91.116.13:4153 , piece_id=BZMRSMICRJUIEDMJHWSI67BQPNPWZAUNBJ3CX5CDGL3FYRD6PMGA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.865749070 (543h5m9.736832125s) [seg_pos={0 0}, upload=476257]: Done begin upload segment.
Dec 28 06:34:09.865787202 (543h5m9.736870247s) [noise=true , address=145.79.227.31:4797 , piece_id=Y5G3XZWKUKJWO4VJ6VVK2KPJPL6DBUNBL2FKDH5KN4TKSYBD6JUA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.866818415 (543h5m9.73790145s) [noise=true , address=145.79.227.45:4229 , piece_id=YTSYDZTA5NPFG7RJD5QAUVBONIVZIYXDLVCSDGFJH2Z3RYBQKR6Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.867255428 (543h5m9.738338473s) [noise=true , address=45.91.116.14:4191 , piece_id=FSGCS4I2W6AXKBAAKIO3H3VVDJCMO2ZJC5I7UTPL3HPWEGFPCH4A, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.867793983 (543h5m9.738877028s) [noise=true , address=77.90.31.19:4842 , piece_id=EPUFZMC6JEEICGDHMTIJ33326OS3Q5LPK6BYCS2MCSLHH4RI2JQQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868206017 (543h5m9.739289052s) [noise=true , address=77.90.31.11:4553 , piece_id=36TL2YGUG55F3AWIBZOLH4NWEOFQAMAFIYI5DYHRZOA7C67AU2BA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868217238 (543h5m9.739300284s) [noise=true , address=145.79.227.11:4729 , piece_id=NOQ6KFZAFIPCILD6R4IIZPADYTODHLMKUCOZOYF6AHFWB3B5CRTA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868668868 (543h5m9.739751904s) [noise=true , address=77.90.31.14:4663 , piece_id=JWAWZOGO35BGX4UXNU7Q726YXBEANHX5T4PITTNXLKSS5FYDXHIA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868732500 (543h5m9.739815535s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869147069 (543h5m9.740230114s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869179722 (543h5m9.740262767s) [noise=true , address=145.79.227.25:4619 , piece_id=QE4ISVK6E4HAJ66OODOSO6BAQBP7BETNWQRU6YYU4J7WW5MK2WGA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869611795 (543h5m9.74069484s) [noise=true , address=145.79.227.16:4911 , piece_id=2YZ4IVX7IEVT2NMS5DPF7VAOYRJFZX2BW2NQ4KCTJG4DS4KS462Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.870118559 (543h5m9.741201604s) [noise=true , address=45.91.116.9:4263 , piece_id=O7RGLAZBP7JGULRDJLYI3F65OMR5BQJDHJBF5YJJY6EVQHDCQH7Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.870684617 (543h5m9.741767672s) [noise=true , address=145.79.227.50:4375 , piece_id=K5GHFVXIHUTT3IQLDKFFFYKVZ24MFJR2RMQVNKKI4BIQW4ZLHEMQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.871095760 (543h5m9.742178805s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.872053514 (543h5m9.743136549s) [noise=true , address=145.79.227.17:4972 , piece_id=WLEHEL6HUTMH4DXZJC3IDYNS3WMVEMILDDSAPL2HQO6SQRJCILFQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:11.740792204 (543h5m11.611875239s) [upload=476257]: Next returned nil segment
Dec 28 06:34:11.740807092 (543h5m11.611890127s) [upload=476257]: Waiting for error group managing segments...
Dec 28 06:34:12.077641418 (543h5m11.948724453s) [noise=true , address=77.90.31.19:4842 , piece_id=EPUFZMC6JEEICGDHMTIJ33326OS3Q5LPK6BYCS2MCSLHH4RI2JQQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.140083399 (543h5m12.011166434s) [noise=true , address=45.91.116.13:4153 , piece_id=BZMRSMICRJUIEDMJHWSI67BQPNPWZAUNBJ3CX5CDGL3FYRD6PMGA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.247241838 (543h5m12.118324883s) [noise=true , address=45.91.116.14:4191 , piece_id=FSGCS4I2W6AXKBAAKIO3H3VVDJCMO2ZJC5I7UTPL3HPWEGFPCH4A, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.281101414 (543h5m12.152184459s) [noise=true , address=77.90.31.11:4553 , piece_id=36TL2YGUG55F3AWIBZOLH4NWEOFQAMAFIYI5DYHRZOA7C67AU2BA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.572990687 (543h5m12.444073732s) [noise=true , address=45.91.116.9:4263 , piece_id=O7RGLAZBP7JGULRDJLYI3F65OMR5BQJDHJBF5YJJY6EVQHDCQH7Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.700970623 (543h5m12.572053658s) [noise=true , address=145.79.227.11:4729 , piece_id=NOQ6KFZAFIPCILD6R4IIZPADYTODHLMKUCOZOYF6AHFWB3B5CRTA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.722390811 (543h5m12.593473846s) [noise=true , address=145.79.227.31:4797 , piece_id=Y5G3XZWKUKJWO4VJ6VVK2KPJPL6DBUNBL2FKDH5KN4TKSYBD6JUA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.751548203 (543h5m12.622631248s) [noise=true , address=145.79.227.45:4229 , piece_id=YTSYDZTA5NPFG7RJD5QAUVBONIVZIYXDLVCSDGFJH2Z3RYBQKR6Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.086115831 (543h5m12.957198866s) [noise=true , address=77.90.31.14:4663 , piece_id=JWAWZOGO35BGX4UXNU7Q726YXBEANHX5T4PITTNXLKSS5FYDXHIA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.088302755 (543h5m12.959385801s) [noise=true , address=145.79.227.50:4375 , piece_id=K5GHFVXIHUTT3IQLDKFFFYKVZ24MFJR2RMQVNKKI4BIQW4ZLHEMQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.163593835 (543h5m13.03467687s) [noise=true , address=145.79.227.25:4619 , piece_id=QE4ISVK6E4HAJ66OODOSO6BAQBP7BETNWQRU6YYU4J7WW5MK2WGA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.180506259 (543h5m13.051589304s) [noise=true , address=145.79.227.17:4972 , piece_id=WLEHEL6HUTMH4DXZJC3IDYNS3WMVEMILDDSAPL2HQO6SQRJCILFQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.280406527 (543h5m13.151489562s) [noise=true , address=145.79.227.16:4911 , piece_id=2YZ4IVX7IEVT2NMS5DPF7VAOYRJFZX2BW2NQ4KCTJG4DS4KS462Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.280435773 (543h5m13.151518818s) [seg_pos={0 0}, upload=476257]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:34:13.280623320 (543h5m13.151706355s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection
Dec 28 06:34:13.280628881 (543h5m13.151711916s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.280789476 (543h5m13.151872511s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection
Dec 28 06:34:13.280804916 (543h5m13.151887961s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.281055182 (543h5m13.152138227s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7): protocol: read tcp 188.166.225.170:55658->145.79.227.52:4446: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 06:34:13.281065091 (543h5m13.152148126s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.281088385 (543h5m13.152171431s) [seg_pos={0 0}, upload=476257]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:34:13.281175762 (543h5m13.152258807s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:34:13.281182455 (543h5m13.15226549s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:34:13.580509800 (543h5m13.451592845s) [upload=476257]: Done uploading segments...
Dec 28 06:34:13.580540228 (543h5m13.451623293s) [upload=476257]: Upload finished. err: <nil>
Duration: 5.922341546s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
35.268717327s trace:
https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6
FAIL (2025-12-28 06:38:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded
======LOGS======
) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment
Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments...
Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces.
Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection
Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments...
Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil>
Duration: 35.268733668s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [..X.X.XX..X...X..XX.X...XX.....X................X.X.....X.X.] duration:
4.862961923s trace:
https://jaeger.dev.storj.io/trace/6af3578081c62393
FAIL (2025-12-28 06:38:24 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.862987131s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [..X..X....X...X..X.XX.XXX..X..X.........X.XX..X....X.XX.....] duration:
6.090789187s trace:
https://jaeger.dev.storj.io/trace/704c6935d5cc4092
FAIL (2025-12-28 06:38:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
h5m9.486928352s) [upload=476257]: Got next segment. Inline: false
Dec 28 06:34:09.615898808 (543h5m9.486981843s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:34:09.615905691 (543h5m9.486988726s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:34:09.865608412 (543h5m9.736691457s) [seg_pos={0 0}, upload=476257]: Begin upload segment...
Dec 28 06:34:09.865736285 (543h5m9.73681933s) [noise=true , address=45.91.116.13:4153 , piece_id=BZMRSMICRJUIEDMJHWSI67BQPNPWZAUNBJ3CX5CDGL3FYRD6PMGA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.865749070 (543h5m9.736832125s) [seg_pos={0 0}, upload=476257]: Done begin upload segment.
Dec 28 06:34:09.865787202 (543h5m9.736870247s) [noise=true , address=145.79.227.31:4797 , piece_id=Y5G3XZWKUKJWO4VJ6VVK2KPJPL6DBUNBL2FKDH5KN4TKSYBD6JUA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.866818415 (543h5m9.73790145s) [noise=true , address=145.79.227.45:4229 , piece_id=YTSYDZTA5NPFG7RJD5QAUVBONIVZIYXDLVCSDGFJH2Z3RYBQKR6Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.867255428 (543h5m9.738338473s) [noise=true , address=45.91.116.14:4191 , piece_id=FSGCS4I2W6AXKBAAKIO3H3VVDJCMO2ZJC5I7UTPL3HPWEGFPCH4A, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.867793983 (543h5m9.738877028s) [noise=true , address=77.90.31.19:4842 , piece_id=EPUFZMC6JEEICGDHMTIJ33326OS3Q5LPK6BYCS2MCSLHH4RI2JQQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868206017 (543h5m9.739289052s) [noise=true , address=77.90.31.11:4553 , piece_id=36TL2YGUG55F3AWIBZOLH4NWEOFQAMAFIYI5DYHRZOA7C67AU2BA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868217238 (543h5m9.739300284s) [noise=true , address=145.79.227.11:4729 , piece_id=NOQ6KFZAFIPCILD6R4IIZPADYTODHLMKUCOZOYF6AHFWB3B5CRTA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868668868 (543h5m9.739751904s) [noise=true , address=77.90.31.14:4663 , piece_id=JWAWZOGO35BGX4UXNU7Q726YXBEANHX5T4PITTNXLKSS5FYDXHIA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868732500 (543h5m9.739815535s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869147069 (543h5m9.740230114s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869179722 (543h5m9.740262767s) [noise=true , address=145.79.227.25:4619 , piece_id=QE4ISVK6E4HAJ66OODOSO6BAQBP7BETNWQRU6YYU4J7WW5MK2WGA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869611795 (543h5m9.74069484s) [noise=true , address=145.79.227.16:4911 , piece_id=2YZ4IVX7IEVT2NMS5DPF7VAOYRJFZX2BW2NQ4KCTJG4DS4KS462Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.870118559 (543h5m9.741201604s) [noise=true , address=45.91.116.9:4263 , piece_id=O7RGLAZBP7JGULRDJLYI3F65OMR5BQJDHJBF5YJJY6EVQHDCQH7Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.870684617 (543h5m9.741767672s) [noise=true , address=145.79.227.50:4375 , piece_id=K5GHFVXIHUTT3IQLDKFFFYKVZ24MFJR2RMQVNKKI4BIQW4ZLHEMQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.871095760 (543h5m9.742178805s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.872053514 (543h5m9.743136549s) [noise=true , address=145.79.227.17:4972 , piece_id=WLEHEL6HUTMH4DXZJC3IDYNS3WMVEMILDDSAPL2HQO6SQRJCILFQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:11.740792204 (543h5m11.611875239s) [upload=476257]: Next returned nil segment
Dec 28 06:34:11.740807092 (543h5m11.611890127s) [upload=476257]: Waiting for error group managing segments...
Dec 28 06:34:12.077641418 (543h5m11.948724453s) [noise=true , address=77.90.31.19:4842 , piece_id=EPUFZMC6JEEICGDHMTIJ33326OS3Q5LPK6BYCS2MCSLHH4RI2JQQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.140083399 (543h5m12.011166434s) [noise=true , address=45.91.116.13:4153 , piece_id=BZMRSMICRJUIEDMJHWSI67BQPNPWZAUNBJ3CX5CDGL3FYRD6PMGA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.247241838 (543h5m12.118324883s) [noise=true , address=45.91.116.14:4191 , piece_id=FSGCS4I2W6AXKBAAKIO3H3VVDJCMO2ZJC5I7UTPL3HPWEGFPCH4A, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.281101414 (543h5m12.152184459s) [noise=true , address=77.90.31.11:4553 , piece_id=36TL2YGUG55F3AWIBZOLH4NWEOFQAMAFIYI5DYHRZOA7C67AU2BA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.572990687 (543h5m12.444073732s) [noise=true , address=45.91.116.9:4263 , piece_id=O7RGLAZBP7JGULRDJLYI3F65OMR5BQJDHJBF5YJJY6EVQHDCQH7Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.700970623 (543h5m12.572053658s) [noise=true , address=145.79.227.11:4729 , piece_id=NOQ6KFZAFIPCILD6R4IIZPADYTODHLMKUCOZOYF6AHFWB3B5CRTA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.722390811 (543h5m12.593473846s) [noise=true , address=145.79.227.31:4797 , piece_id=Y5G3XZWKUKJWO4VJ6VVK2KPJPL6DBUNBL2FKDH5KN4TKSYBD6JUA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.751548203 (543h5m12.622631248s) [noise=true , address=145.79.227.45:4229 , piece_id=YTSYDZTA5NPFG7RJD5QAUVBONIVZIYXDLVCSDGFJH2Z3RYBQKR6Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.086115831 (543h5m12.957198866s) [noise=true , address=77.90.31.14:4663 , piece_id=JWAWZOGO35BGX4UXNU7Q726YXBEANHX5T4PITTNXLKSS5FYDXHIA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.088302755 (543h5m12.959385801s) [noise=true , address=145.79.227.50:4375 , piece_id=K5GHFVXIHUTT3IQLDKFFFYKVZ24MFJR2RMQVNKKI4BIQW4ZLHEMQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.163593835 (543h5m13.03467687s) [noise=true , address=145.79.227.25:4619 , piece_id=QE4ISVK6E4HAJ66OODOSO6BAQBP7BETNWQRU6YYU4J7WW5MK2WGA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.180506259 (543h5m13.051589304s) [noise=true , address=145.79.227.17:4972 , piece_id=WLEHEL6HUTMH4DXZJC3IDYNS3WMVEMILDDSAPL2HQO6SQRJCILFQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.280406527 (543h5m13.151489562s) [noise=true , address=145.79.227.16:4911 , piece_id=2YZ4IVX7IEVT2NMS5DPF7VAOYRJFZX2BW2NQ4KCTJG4DS4KS462Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.280435773 (543h5m13.151518818s) [seg_pos={0 0}, upload=476257]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:34:13.280623320 (543h5m13.151706355s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection
Dec 28 06:34:13.280628881 (543h5m13.151711916s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.280789476 (543h5m13.151872511s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection
Dec 28 06:34:13.280804916 (543h5m13.151887961s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.281055182 (543h5m13.152138227s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7): protocol: read tcp 188.166.225.170:55658->145.79.227.52:4446: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 06:34:13.281065091 (543h5m13.152148126s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.281088385 (543h5m13.152171431s) [seg_pos={0 0}, upload=476257]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:34:13.281175762 (543h5m13.152258807s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:34:13.281182455 (543h5m13.15226549s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:34:13.580509800 (543h5m13.451592845s) [upload=476257]: Done uploading segments...
Dec 28 06:34:13.580540228 (543h5m13.451623293s) [upload=476257]: Upload finished. err: <nil>
Duration: 6.090820026s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
40.748267941s trace:
https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6
FAIL (2025-12-28 06:38:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded
======LOGS======
) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment
Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments...
Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces.
Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection
Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments...
Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil>
Duration: 40.748284183s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [...X.X.XX..X...X..XX.X...XX.....X................X.X.....X.X] duration:
5.001605103s trace:
https://jaeger.dev.storj.io/trace/6af3578081c62393
FAIL (2025-12-28 06:38:04 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001624009s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [X...X.X.XX..X...X..XX.X...XX.....X................X.X.....X.] duration:
5.0010622s trace:
https://jaeger.dev.storj.io/trace/8d178805925806d
FAIL (2025-12-28 06:37:38 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001089882s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [...X..X....X...X..X.XX.XXX..X..X.........X.XX..X....X.XX....] duration:
5.691591225s trace:
https://jaeger.dev.storj.io/trace/704c6935d5cc4092
FAIL (2025-12-28 06:37:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
h5m9.486928352s) [upload=476257]: Got next segment. Inline: false
Dec 28 06:34:09.615898808 (543h5m9.486981843s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:34:09.615905691 (543h5m9.486988726s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:34:09.865608412 (543h5m9.736691457s) [seg_pos={0 0}, upload=476257]: Begin upload segment...
Dec 28 06:34:09.865736285 (543h5m9.73681933s) [noise=true , address=45.91.116.13:4153 , piece_id=BZMRSMICRJUIEDMJHWSI67BQPNPWZAUNBJ3CX5CDGL3FYRD6PMGA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.865749070 (543h5m9.736832125s) [seg_pos={0 0}, upload=476257]: Done begin upload segment.
Dec 28 06:34:09.865787202 (543h5m9.736870247s) [noise=true , address=145.79.227.31:4797 , piece_id=Y5G3XZWKUKJWO4VJ6VVK2KPJPL6DBUNBL2FKDH5KN4TKSYBD6JUA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.866818415 (543h5m9.73790145s) [noise=true , address=145.79.227.45:4229 , piece_id=YTSYDZTA5NPFG7RJD5QAUVBONIVZIYXDLVCSDGFJH2Z3RYBQKR6Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.867255428 (543h5m9.738338473s) [noise=true , address=45.91.116.14:4191 , piece_id=FSGCS4I2W6AXKBAAKIO3H3VVDJCMO2ZJC5I7UTPL3HPWEGFPCH4A, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.867793983 (543h5m9.738877028s) [noise=true , address=77.90.31.19:4842 , piece_id=EPUFZMC6JEEICGDHMTIJ33326OS3Q5LPK6BYCS2MCSLHH4RI2JQQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868206017 (543h5m9.739289052s) [noise=true , address=77.90.31.11:4553 , piece_id=36TL2YGUG55F3AWIBZOLH4NWEOFQAMAFIYI5DYHRZOA7C67AU2BA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868217238 (543h5m9.739300284s) [noise=true , address=145.79.227.11:4729 , piece_id=NOQ6KFZAFIPCILD6R4IIZPADYTODHLMKUCOZOYF6AHFWB3B5CRTA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868668868 (543h5m9.739751904s) [noise=true , address=77.90.31.14:4663 , piece_id=JWAWZOGO35BGX4UXNU7Q726YXBEANHX5T4PITTNXLKSS5FYDXHIA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868732500 (543h5m9.739815535s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869147069 (543h5m9.740230114s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869179722 (543h5m9.740262767s) [noise=true , address=145.79.227.25:4619 , piece_id=QE4ISVK6E4HAJ66OODOSO6BAQBP7BETNWQRU6YYU4J7WW5MK2WGA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869611795 (543h5m9.74069484s) [noise=true , address=145.79.227.16:4911 , piece_id=2YZ4IVX7IEVT2NMS5DPF7VAOYRJFZX2BW2NQ4KCTJG4DS4KS462Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.870118559 (543h5m9.741201604s) [noise=true , address=45.91.116.9:4263 , piece_id=O7RGLAZBP7JGULRDJLYI3F65OMR5BQJDHJBF5YJJY6EVQHDCQH7Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.870684617 (543h5m9.741767672s) [noise=true , address=145.79.227.50:4375 , piece_id=K5GHFVXIHUTT3IQLDKFFFYKVZ24MFJR2RMQVNKKI4BIQW4ZLHEMQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.871095760 (543h5m9.742178805s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.872053514 (543h5m9.743136549s) [noise=true , address=145.79.227.17:4972 , piece_id=WLEHEL6HUTMH4DXZJC3IDYNS3WMVEMILDDSAPL2HQO6SQRJCILFQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:11.740792204 (543h5m11.611875239s) [upload=476257]: Next returned nil segment
Dec 28 06:34:11.740807092 (543h5m11.611890127s) [upload=476257]: Waiting for error group managing segments...
Dec 28 06:34:12.077641418 (543h5m11.948724453s) [noise=true , address=77.90.31.19:4842 , piece_id=EPUFZMC6JEEICGDHMTIJ33326OS3Q5LPK6BYCS2MCSLHH4RI2JQQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.140083399 (543h5m12.011166434s) [noise=true , address=45.91.116.13:4153 , piece_id=BZMRSMICRJUIEDMJHWSI67BQPNPWZAUNBJ3CX5CDGL3FYRD6PMGA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.247241838 (543h5m12.118324883s) [noise=true , address=45.91.116.14:4191 , piece_id=FSGCS4I2W6AXKBAAKIO3H3VVDJCMO2ZJC5I7UTPL3HPWEGFPCH4A, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.281101414 (543h5m12.152184459s) [noise=true , address=77.90.31.11:4553 , piece_id=36TL2YGUG55F3AWIBZOLH4NWEOFQAMAFIYI5DYHRZOA7C67AU2BA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.572990687 (543h5m12.444073732s) [noise=true , address=45.91.116.9:4263 , piece_id=O7RGLAZBP7JGULRDJLYI3F65OMR5BQJDHJBF5YJJY6EVQHDCQH7Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.700970623 (543h5m12.572053658s) [noise=true , address=145.79.227.11:4729 , piece_id=NOQ6KFZAFIPCILD6R4IIZPADYTODHLMKUCOZOYF6AHFWB3B5CRTA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.722390811 (543h5m12.593473846s) [noise=true , address=145.79.227.31:4797 , piece_id=Y5G3XZWKUKJWO4VJ6VVK2KPJPL6DBUNBL2FKDH5KN4TKSYBD6JUA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.751548203 (543h5m12.622631248s) [noise=true , address=145.79.227.45:4229 , piece_id=YTSYDZTA5NPFG7RJD5QAUVBONIVZIYXDLVCSDGFJH2Z3RYBQKR6Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.086115831 (543h5m12.957198866s) [noise=true , address=77.90.31.14:4663 , piece_id=JWAWZOGO35BGX4UXNU7Q726YXBEANHX5T4PITTNXLKSS5FYDXHIA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.088302755 (543h5m12.959385801s) [noise=true , address=145.79.227.50:4375 , piece_id=K5GHFVXIHUTT3IQLDKFFFYKVZ24MFJR2RMQVNKKI4BIQW4ZLHEMQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.163593835 (543h5m13.03467687s) [noise=true , address=145.79.227.25:4619 , piece_id=QE4ISVK6E4HAJ66OODOSO6BAQBP7BETNWQRU6YYU4J7WW5MK2WGA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.180506259 (543h5m13.051589304s) [noise=true , address=145.79.227.17:4972 , piece_id=WLEHEL6HUTMH4DXZJC3IDYNS3WMVEMILDDSAPL2HQO6SQRJCILFQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.280406527 (543h5m13.151489562s) [noise=true , address=145.79.227.16:4911 , piece_id=2YZ4IVX7IEVT2NMS5DPF7VAOYRJFZX2BW2NQ4KCTJG4DS4KS462Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.280435773 (543h5m13.151518818s) [seg_pos={0 0}, upload=476257]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:34:13.280623320 (543h5m13.151706355s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection
Dec 28 06:34:13.280628881 (543h5m13.151711916s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.280789476 (543h5m13.151872511s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection
Dec 28 06:34:13.280804916 (543h5m13.151887961s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.281055182 (543h5m13.152138227s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7): protocol: read tcp 188.166.225.170:55658->145.79.227.52:4446: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 06:34:13.281065091 (543h5m13.152148126s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.281088385 (543h5m13.152171431s) [seg_pos={0 0}, upload=476257]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:34:13.281175762 (543h5m13.152258807s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:34:13.281182455 (543h5m13.15226549s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:34:13.580509800 (543h5m13.451592845s) [upload=476257]: Done uploading segments...
Dec 28 06:34:13.580540228 (543h5m13.451623293s) [upload=476257]: Upload finished. err: <nil>
Duration: 5.691619569s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [.X...X.X.XX..X...X..XX.X...XX.....X................X.X.....X] duration:
5.001437826s trace:
https://jaeger.dev.storj.io/trace/8d178805925806d
FAIL (2025-12-28 06:37:04 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00145051s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
28.334108389s trace:
https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6
FAIL (2025-12-28 06:36:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded
======LOGS======
) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment
Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments...
Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces.
Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection
Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments...
Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil>
Duration: 28.334155068s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [X.X...X.X.XX..X...X..XX.X...XX.....X................X.X.....] duration:
4.383647777s trace:
https://jaeger.dev.storj.io/trace/5a300537fc10f39f
FAIL (2025-12-28 06:36:37 UTC)
orbiter warning: context deadline exceeded
Duration: 4.383670031s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [X...X..X....X...X..X.XX.XXX..X..X.........X.XX..X....X.XX...] duration:
6.003222805s trace:
https://jaeger.dev.storj.io/trace/704c6935d5cc4092
FAIL (2025-12-28 06:36:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
h5m9.486928352s) [upload=476257]: Got next segment. Inline: false
Dec 28 06:34:09.615898808 (543h5m9.486981843s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:34:09.615905691 (543h5m9.486988726s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:34:09.865608412 (543h5m9.736691457s) [seg_pos={0 0}, upload=476257]: Begin upload segment...
Dec 28 06:34:09.865736285 (543h5m9.73681933s) [noise=true , address=45.91.116.13:4153 , piece_id=BZMRSMICRJUIEDMJHWSI67BQPNPWZAUNBJ3CX5CDGL3FYRD6PMGA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.865749070 (543h5m9.736832125s) [seg_pos={0 0}, upload=476257]: Done begin upload segment.
Dec 28 06:34:09.865787202 (543h5m9.736870247s) [noise=true , address=145.79.227.31:4797 , piece_id=Y5G3XZWKUKJWO4VJ6VVK2KPJPL6DBUNBL2FKDH5KN4TKSYBD6JUA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.866818415 (543h5m9.73790145s) [noise=true , address=145.79.227.45:4229 , piece_id=YTSYDZTA5NPFG7RJD5QAUVBONIVZIYXDLVCSDGFJH2Z3RYBQKR6Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.867255428 (543h5m9.738338473s) [noise=true , address=45.91.116.14:4191 , piece_id=FSGCS4I2W6AXKBAAKIO3H3VVDJCMO2ZJC5I7UTPL3HPWEGFPCH4A, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.867793983 (543h5m9.738877028s) [noise=true , address=77.90.31.19:4842 , piece_id=EPUFZMC6JEEICGDHMTIJ33326OS3Q5LPK6BYCS2MCSLHH4RI2JQQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868206017 (543h5m9.739289052s) [noise=true , address=77.90.31.11:4553 , piece_id=36TL2YGUG55F3AWIBZOLH4NWEOFQAMAFIYI5DYHRZOA7C67AU2BA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868217238 (543h5m9.739300284s) [noise=true , address=145.79.227.11:4729 , piece_id=NOQ6KFZAFIPCILD6R4IIZPADYTODHLMKUCOZOYF6AHFWB3B5CRTA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868668868 (543h5m9.739751904s) [noise=true , address=77.90.31.14:4663 , piece_id=JWAWZOGO35BGX4UXNU7Q726YXBEANHX5T4PITTNXLKSS5FYDXHIA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868732500 (543h5m9.739815535s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869147069 (543h5m9.740230114s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869179722 (543h5m9.740262767s) [noise=true , address=145.79.227.25:4619 , piece_id=QE4ISVK6E4HAJ66OODOSO6BAQBP7BETNWQRU6YYU4J7WW5MK2WGA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869611795 (543h5m9.74069484s) [noise=true , address=145.79.227.16:4911 , piece_id=2YZ4IVX7IEVT2NMS5DPF7VAOYRJFZX2BW2NQ4KCTJG4DS4KS462Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.870118559 (543h5m9.741201604s) [noise=true , address=45.91.116.9:4263 , piece_id=O7RGLAZBP7JGULRDJLYI3F65OMR5BQJDHJBF5YJJY6EVQHDCQH7Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.870684617 (543h5m9.741767672s) [noise=true , address=145.79.227.50:4375 , piece_id=K5GHFVXIHUTT3IQLDKFFFYKVZ24MFJR2RMQVNKKI4BIQW4ZLHEMQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.871095760 (543h5m9.742178805s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.872053514 (543h5m9.743136549s) [noise=true , address=145.79.227.17:4972 , piece_id=WLEHEL6HUTMH4DXZJC3IDYNS3WMVEMILDDSAPL2HQO6SQRJCILFQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:11.740792204 (543h5m11.611875239s) [upload=476257]: Next returned nil segment
Dec 28 06:34:11.740807092 (543h5m11.611890127s) [upload=476257]: Waiting for error group managing segments...
Dec 28 06:34:12.077641418 (543h5m11.948724453s) [noise=true , address=77.90.31.19:4842 , piece_id=EPUFZMC6JEEICGDHMTIJ33326OS3Q5LPK6BYCS2MCSLHH4RI2JQQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.140083399 (543h5m12.011166434s) [noise=true , address=45.91.116.13:4153 , piece_id=BZMRSMICRJUIEDMJHWSI67BQPNPWZAUNBJ3CX5CDGL3FYRD6PMGA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.247241838 (543h5m12.118324883s) [noise=true , address=45.91.116.14:4191 , piece_id=FSGCS4I2W6AXKBAAKIO3H3VVDJCMO2ZJC5I7UTPL3HPWEGFPCH4A, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.281101414 (543h5m12.152184459s) [noise=true , address=77.90.31.11:4553 , piece_id=36TL2YGUG55F3AWIBZOLH4NWEOFQAMAFIYI5DYHRZOA7C67AU2BA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.572990687 (543h5m12.444073732s) [noise=true , address=45.91.116.9:4263 , piece_id=O7RGLAZBP7JGULRDJLYI3F65OMR5BQJDHJBF5YJJY6EVQHDCQH7Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.700970623 (543h5m12.572053658s) [noise=true , address=145.79.227.11:4729 , piece_id=NOQ6KFZAFIPCILD6R4IIZPADYTODHLMKUCOZOYF6AHFWB3B5CRTA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.722390811 (543h5m12.593473846s) [noise=true , address=145.79.227.31:4797 , piece_id=Y5G3XZWKUKJWO4VJ6VVK2KPJPL6DBUNBL2FKDH5KN4TKSYBD6JUA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.751548203 (543h5m12.622631248s) [noise=true , address=145.79.227.45:4229 , piece_id=YTSYDZTA5NPFG7RJD5QAUVBONIVZIYXDLVCSDGFJH2Z3RYBQKR6Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.086115831 (543h5m12.957198866s) [noise=true , address=77.90.31.14:4663 , piece_id=JWAWZOGO35BGX4UXNU7Q726YXBEANHX5T4PITTNXLKSS5FYDXHIA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.088302755 (543h5m12.959385801s) [noise=true , address=145.79.227.50:4375 , piece_id=K5GHFVXIHUTT3IQLDKFFFYKVZ24MFJR2RMQVNKKI4BIQW4ZLHEMQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.163593835 (543h5m13.03467687s) [noise=true , address=145.79.227.25:4619 , piece_id=QE4ISVK6E4HAJ66OODOSO6BAQBP7BETNWQRU6YYU4J7WW5MK2WGA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.180506259 (543h5m13.051589304s) [noise=true , address=145.79.227.17:4972 , piece_id=WLEHEL6HUTMH4DXZJC3IDYNS3WMVEMILDDSAPL2HQO6SQRJCILFQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.280406527 (543h5m13.151489562s) [noise=true , address=145.79.227.16:4911 , piece_id=2YZ4IVX7IEVT2NMS5DPF7VAOYRJFZX2BW2NQ4KCTJG4DS4KS462Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.280435773 (543h5m13.151518818s) [seg_pos={0 0}, upload=476257]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:34:13.280623320 (543h5m13.151706355s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection
Dec 28 06:34:13.280628881 (543h5m13.151711916s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.280789476 (543h5m13.151872511s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection
Dec 28 06:34:13.280804916 (543h5m13.151887961s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.281055182 (543h5m13.152138227s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7): protocol: read tcp 188.166.225.170:55658->145.79.227.52:4446: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 06:34:13.281065091 (543h5m13.152148126s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.281088385 (543h5m13.152171431s) [seg_pos={0 0}, upload=476257]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:34:13.281175762 (543h5m13.152258807s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:34:13.281182455 (543h5m13.15226549s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:34:13.580509800 (543h5m13.451592845s) [upload=476257]: Done uploading segments...
Dec 28 06:34:13.580540228 (543h5m13.451623293s) [upload=476257]: Upload finished. err: <nil>
Duration: 6.003248714s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
39.065128207s trace:
https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6
FAIL (2025-12-28 06:36:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded
======LOGS======
) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment
Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments...
Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces.
Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection
Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments...
Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil>
Duration: 39.065144107s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
17 out of 60 is failed (threshold=6) [XX.X...X.X.XX..X...X..XX.X...XX.....X................X.X....] duration:
3.36434971s trace:
https://jaeger.dev.storj.io/trace/5a300537fc10f39f
FAIL (2025-12-28 06:36:19 UTC)
orbiter warning: context deadline exceeded
Duration: 3.364394705s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [.X...X..X....X...X..X.XX.XXX..X..X.........X.XX..X....X.XX..] duration:
26.334506086s trace:
https://jaeger.dev.storj.io/trace/704c6935d5cc4092
FAIL (2025-12-28 06:36:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
h5m9.486928352s) [upload=476257]: Got next segment. Inline: false
Dec 28 06:34:09.615898808 (543h5m9.486981843s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:34:09.615905691 (543h5m9.486988726s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:34:09.865608412 (543h5m9.736691457s) [seg_pos={0 0}, upload=476257]: Begin upload segment...
Dec 28 06:34:09.865736285 (543h5m9.73681933s) [noise=true , address=45.91.116.13:4153 , piece_id=BZMRSMICRJUIEDMJHWSI67BQPNPWZAUNBJ3CX5CDGL3FYRD6PMGA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.865749070 (543h5m9.736832125s) [seg_pos={0 0}, upload=476257]: Done begin upload segment.
Dec 28 06:34:09.865787202 (543h5m9.736870247s) [noise=true , address=145.79.227.31:4797 , piece_id=Y5G3XZWKUKJWO4VJ6VVK2KPJPL6DBUNBL2FKDH5KN4TKSYBD6JUA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.866818415 (543h5m9.73790145s) [noise=true , address=145.79.227.45:4229 , piece_id=YTSYDZTA5NPFG7RJD5QAUVBONIVZIYXDLVCSDGFJH2Z3RYBQKR6Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.867255428 (543h5m9.738338473s) [noise=true , address=45.91.116.14:4191 , piece_id=FSGCS4I2W6AXKBAAKIO3H3VVDJCMO2ZJC5I7UTPL3HPWEGFPCH4A, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.867793983 (543h5m9.738877028s) [noise=true , address=77.90.31.19:4842 , piece_id=EPUFZMC6JEEICGDHMTIJ33326OS3Q5LPK6BYCS2MCSLHH4RI2JQQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868206017 (543h5m9.739289052s) [noise=true , address=77.90.31.11:4553 , piece_id=36TL2YGUG55F3AWIBZOLH4NWEOFQAMAFIYI5DYHRZOA7C67AU2BA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868217238 (543h5m9.739300284s) [noise=true , address=145.79.227.11:4729 , piece_id=NOQ6KFZAFIPCILD6R4IIZPADYTODHLMKUCOZOYF6AHFWB3B5CRTA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868668868 (543h5m9.739751904s) [noise=true , address=77.90.31.14:4663 , piece_id=JWAWZOGO35BGX4UXNU7Q726YXBEANHX5T4PITTNXLKSS5FYDXHIA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.868732500 (543h5m9.739815535s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869147069 (543h5m9.740230114s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869179722 (543h5m9.740262767s) [noise=true , address=145.79.227.25:4619 , piece_id=QE4ISVK6E4HAJ66OODOSO6BAQBP7BETNWQRU6YYU4J7WW5MK2WGA, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.869611795 (543h5m9.74069484s) [noise=true , address=145.79.227.16:4911 , piece_id=2YZ4IVX7IEVT2NMS5DPF7VAOYRJFZX2BW2NQ4KCTJG4DS4KS462Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.870118559 (543h5m9.741201604s) [noise=true , address=45.91.116.9:4263 , piece_id=O7RGLAZBP7JGULRDJLYI3F65OMR5BQJDHJBF5YJJY6EVQHDCQH7Q, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.870684617 (543h5m9.741767672s) [noise=true , address=145.79.227.50:4375 , piece_id=K5GHFVXIHUTT3IQLDKFFFYKVZ24MFJR2RMQVNKKI4BIQW4ZLHEMQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.871095760 (543h5m9.742178805s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:09.872053514 (543h5m9.743136549s) [noise=true , address=145.79.227.17:4972 , piece_id=WLEHEL6HUTMH4DXZJC3IDYNS3WMVEMILDDSAPL2HQO6SQRJCILFQ, seg_pos={0 0}, upload=476257]: Uploading piece...
Dec 28 06:34:11.740792204 (543h5m11.611875239s) [upload=476257]: Next returned nil segment
Dec 28 06:34:11.740807092 (543h5m11.611890127s) [upload=476257]: Waiting for error group managing segments...
Dec 28 06:34:12.077641418 (543h5m11.948724453s) [noise=true , address=77.90.31.19:4842 , piece_id=EPUFZMC6JEEICGDHMTIJ33326OS3Q5LPK6BYCS2MCSLHH4RI2JQQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.140083399 (543h5m12.011166434s) [noise=true , address=45.91.116.13:4153 , piece_id=BZMRSMICRJUIEDMJHWSI67BQPNPWZAUNBJ3CX5CDGL3FYRD6PMGA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.247241838 (543h5m12.118324883s) [noise=true , address=45.91.116.14:4191 , piece_id=FSGCS4I2W6AXKBAAKIO3H3VVDJCMO2ZJC5I7UTPL3HPWEGFPCH4A, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.281101414 (543h5m12.152184459s) [noise=true , address=77.90.31.11:4553 , piece_id=36TL2YGUG55F3AWIBZOLH4NWEOFQAMAFIYI5DYHRZOA7C67AU2BA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.572990687 (543h5m12.444073732s) [noise=true , address=45.91.116.9:4263 , piece_id=O7RGLAZBP7JGULRDJLYI3F65OMR5BQJDHJBF5YJJY6EVQHDCQH7Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.700970623 (543h5m12.572053658s) [noise=true , address=145.79.227.11:4729 , piece_id=NOQ6KFZAFIPCILD6R4IIZPADYTODHLMKUCOZOYF6AHFWB3B5CRTA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.722390811 (543h5m12.593473846s) [noise=true , address=145.79.227.31:4797 , piece_id=Y5G3XZWKUKJWO4VJ6VVK2KPJPL6DBUNBL2FKDH5KN4TKSYBD6JUA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:12.751548203 (543h5m12.622631248s) [noise=true , address=145.79.227.45:4229 , piece_id=YTSYDZTA5NPFG7RJD5QAUVBONIVZIYXDLVCSDGFJH2Z3RYBQKR6Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.086115831 (543h5m12.957198866s) [noise=true , address=77.90.31.14:4663 , piece_id=JWAWZOGO35BGX4UXNU7Q726YXBEANHX5T4PITTNXLKSS5FYDXHIA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.088302755 (543h5m12.959385801s) [noise=true , address=145.79.227.50:4375 , piece_id=K5GHFVXIHUTT3IQLDKFFFYKVZ24MFJR2RMQVNKKI4BIQW4ZLHEMQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.163593835 (543h5m13.03467687s) [noise=true , address=145.79.227.25:4619 , piece_id=QE4ISVK6E4HAJ66OODOSO6BAQBP7BETNWQRU6YYU4J7WW5MK2WGA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.180506259 (543h5m13.051589304s) [noise=true , address=145.79.227.17:4972 , piece_id=WLEHEL6HUTMH4DXZJC3IDYNS3WMVEMILDDSAPL2HQO6SQRJCILFQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.280406527 (543h5m13.151489562s) [noise=true , address=145.79.227.16:4911 , piece_id=2YZ4IVX7IEVT2NMS5DPF7VAOYRJFZX2BW2NQ4KCTJG4DS4KS462Q, seg_pos={0 0}, upload=476257]: Done uploading piece. err: <nil>
Dec 28 06:34:13.280435773 (543h5m13.151518818s) [seg_pos={0 0}, upload=476257]: Segment reached optimal threshold of 13 pieces.
Dec 28 06:34:13.280623320 (543h5m13.151706355s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection; write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37910->145.79.227.5:4221: use of closed network connection
Dec 28 06:34:13.280628881 (543h5m13.151711916s) [noise=true , address=145.79.227.5:4221 , piece_id=T4REKSYCWINF2O67ZOKYXSTD6TGNWNRHN2UD5NVOHJ3GOWKIGUFA, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.280789476 (543h5m13.151872511s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41046->45.91.116.11:4118: use of closed network connection
Dec 28 06:34:13.280804916 (543h5m13.151887961s) [noise=true , address=45.91.116.11:4118 , piece_id=J2MYV4U2IFTALZPRSEI6EYGFXXYUWZXE5OOPH5KFPGDGSMZEW7MA, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.281055182 (543h5m13.152138227s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7): protocol: read tcp 188.166.225.170:55658->145.79.227.52:4446: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 06:34:13.281065091 (543h5m13.152148126s) [noise=true , address=145.79.227.52:4446 , piece_id=WT4IKBG7CLCZGSVVOIXYPSMLTVXDDHGBZ2A75IQINUS45L7NGMZQ, seg_pos={0 0}, upload=476257]: Long tail cancellation due to optimal threshold
Dec 28 06:34:13.281088385 (543h5m13.152171431s) [seg_pos={0 0}, upload=476257]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 06:34:13.281175762 (543h5m13.152258807s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 06:34:13.281182455 (543h5m13.15226549s) [upload=476257]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 06:34:13.580509800 (543h5m13.451592845s) [upload=476257]: Done uploading segments...
Dec 28 06:34:13.580540228 (543h5m13.451623293s) [upload=476257]: Upload finished. err: <nil>
Duration: 26.334542094s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
40.149796198s trace:
https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6
FAIL (2025-12-28 06:35:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded
======LOGS======
) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment
Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments...
Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces.
Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection
Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments...
Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil>
Duration: 40.149827969s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
17 out of 60 is failed (threshold=6) [.XX.X...X.X.XX..X...X..XX.X...XX.....X................X.X...] duration:
3.495532824s trace:
https://jaeger.dev.storj.io/trace/5a300537fc10f39f
FAIL (2025-12-28 06:35:34 UTC)
orbiter warning: context deadline exceeded
Duration: 3.495771238s
core
uplink
us-select-1
us1-placement-20
|